Who's Right?

One of the longest running debates in the digital world is over audience measurement methodologies. Despite the fact that advertisers and their agencies largely discount Web site server-based claims of the number of unique visitors in favor of panel-based measurement of audiences, some site operators still publicly complain about panel-based measurement understating their server counts of unique visitors by a factor of two to three times.

In June 2007, Comscore published a definitive white paper concluding that most of the difference is explained by Internet users’ deleting the cookies placed on their computers by Web site servers -- which is the way many site operators try to keep a count of their unique visitors. Other factors leading to the over-counting by site servers are the inclusion of spider and bot traffic, inaccurate filtering of international traffic, the dynamic of the same person using different computers (e.g. work and home computers) to visit the same site and being counted by the site server as two different visitors, and the fact that about 12% of Internet users’ computers are set to always reject cookies (in these cases, every time such a computer visits a site it is counted as a new unique visitor).

The purpose of this post is not to rehash past arguments but to share some new data that further supports the conclusion that Web site server data inflate the counts of unique visitors.

In the last few months, two new measurement approaches, Google Ad Planner and Quantcast, have been announced, both of which provide an estimate of the size of Web site audiences. Although the methodologies behind both are somewhat opaque, we were curious to see what their measurement shows and to see if they closed the gap between the panel approach and site server data.

The implication behind the Google introduction was that with all the data at its disposal, Google should be able to produce accurate site audience numbers. Although the origin of Quantcast’s numbers is less clear, they claim to combine Web site server data with “panel” browsing data to provide “the best of both worlds” in an audience measurement solution.

Comscore has studied the data from both sources and compared them to Comscore’s own estimates on a site-by-site basis. Among the findings is that neither methodology closes the gap between Comscore’s panel-based audience projections and Web site server numbers. In fact, both approaches generally report audiences that are smaller than Comscore’s. In a comparison of more than 10,000 sites, the average audience reported by Quantcast is 3 percent lower than the average Comscore data for the same sites and Google Ad Planner comes in 44 percent lower. For the purpose of this analysis, we excluded adult sites and promotional server traffic.

Thus, these two new approaches from Quantcast and Google are not providing estimates of the number of unique Web site visitors that are any higher than Comscore’s balanced panel approach. Put another way, there is no evidence from these two independent and alternative methodologies that Comscore’s panel data are undercounting the size of site audiences relative to server data.

At Comscore, we believe that panel-based audience measurement reports are accurate and reliable and the only way to truly count the number of people visiting Web sites. Not cookies and not machines but people – the key audience metric that advertisers and their agencies need when formulating media plans.

We have two additional studies that support the “accuracy” of Comscore’s data. One is a study of the number of ad impressions that Comscore counts through its panel methodology compared to the number of ads actually served by ad servers. The first study compares Comscore data to the ad server logs for twenty different ad campaigns and found that Comscore reported ad impression levels at 97 percent of what the servers showed with a 99 percent correlation. Not too shabby.

The second study - and probably the best testament to the accuracy of the Comscore methodology - is our long-running measurement of e-commerce sales on the Internet. Shortly after the end of each quarter we report our estimate of online sales. When the Department of Commerce issues their report, usually about two months later, the Comscore numbers have been consistently within a few percent of the government estimates — and this has been the case over a period of seven years.

So, who do you think is right? At Comscore, we believe that Web site server numbers for site audiences are too high, for the reasons I have put forth. No other measurement methodology, no matter how sourced or constructed, agrees with the high levels reported by server side measurement or comes even close to bridging the commonly reported gap of 2 to 3x. In fact, the most recent methodologies show similar or lower average audience estimates compared to Comscore. We now have converging evidence confirming that the Comscore numbers are unbiased on average. It seems clear that the gaps with site server numbers are due to an inflationary bias contributed by a variety of problems plaguing the server-based methodology.