A presence everywhere. Photographer: Kim White/Bloomberg
A presence everywhere. Photographer: Kim White/Bloomberg

A few weeks back, I wrote a piece asking whether online ads do any good at all. I was citing a paper that had done experiments on the effectiveness of Google ads. Jim Manzi, who runs a whole lot of experiments for companies, writes to offer his perspective:

Megan McArdle laments that online journalism may be going the way of print journalism. In doing so, she cites a research paper for a series of randomized field experiments at eBay showing that a set of paid search buys had causal effects on sales that were a small fraction of those estimated by nonexperimental methods, and that in certain important cases had no measurable effect on sales at all.

I am always heartened to see controlled experiments take some of the air out of claims derived from nonexperimental methods. The eBay experiments seem from the paper to be well-designed and analyzed, but I don't think that the implications of this are as broad as it might seem.

Imagine measuring the return on investment of television advertising in 1990 by comparing total revenue achieved through the TV as a device with the cost of the ad. Because no material commerce was transacted via your TV, you would always have zero incremental revenue, and therefore terrible ROI. Of course, you would never do this calculation, as it was clear that TV acted as a pure advertising medium, and the transaction media where the causal benefits of the ad showed up were stores, call centers, direct sales forces and so forth.

Online commerce is both an advertising medium and a transaction medium. For some kinds of businesses (like eBay), it is reasonable to assume that all of the incremental revenue caused by an online ad (such as a search heavy-up, a high concentration of ads in a given time frame) will be captured in the online channel. But for many large advertisers, the online ad will also cause sales effects offline (potentially both positive and negative effects).

What is your guess for e-commerce as a proportion of U.S. retail sales? According to the Census Bureau, it's about 6 percent. That is, 94 percent of retail commerce occurs offline. These multichannel effects are a huge deal, because very few of the largest advertisers in America transact purely, or even predominantly, online.

In partnership with Google, my company Applied Predictive Technologies designed and analyzed 15 randomized field experiments between 2008 and 2011 to test the causal impact of search ad buys for multichannel marketers. These were very similar to the eBay experiments: A randomized set of geographic markets was exposed to higher or lower levels of online search ads, then compared with a matched set of control markets to evaluate impact. The big difference is that these 15 experiments evaluated sales impacts on all sales across both online and offline channels.

The data were released to an academic who conducted a meta-analysis of them. The results are eye-opening. Twelve of these 15 experiments demonstrated statistically significant causal impacts on offline sales. These impacts were also economically significant. Because the vast preponderance of consumer commerce happens offline, the total offline margin dollar impacts often dwarf those that occur online.

Unless you think that markets have somehow completely failed, and the enormous growth in online advertising spending over more than a decade has been a product of some kind of mass delusion, this makes sense. Online advertising is not the always-and-everywhere-sure-thing bet that publishers can sometimes claim, but is often very economically effective -- when you carefully measure effects across all channels.

To contact the author of this article: Megan McArdle at mmcardle3@bloomberg.net.

To contact the editor responsible for this article: James Gibney at jgibney5@bloomberg.net.