Comscore Cookie Deletion Study and its Implications for Internet Audience Measurement

Because of this, we can see that cookie-based measurement techniques, whether using Web site server logs or web analytics software, can over-represent the number of unique visitors to a site by a factor of 2.5x. For example, cookie-based measurement will count one typical site visitor who deleted cookies 5 times as 6 different visitors. Our study clearly validates what experts have been saying for years: to get accurate unique visitor counts, you need to track people, not cookies.

Some of the early promises of digital media compared to traditional media like TV and radio were measurability and accountability. And there certainly are examples where the promise was delivered. For starters, digital media can provide accurate counts of advertising impressions, a feat that has never been accomplished by traditional media. In addition, the impact of direct response advertising (e.g. credit card solicitations) can be measured accurately through click-through rates. These are huge advantages for digital media.

However, when one considers other types of measurement, the early promises have proven elusive. Even basic reach and frequency estimates which have been routinely delivered by ad servers are now shown by our study to be flawed, with reach inflated and frequency understated. The Internet industry has also found the impact of brand-building advertising harder to measure and certainly not as simple as counting click-through rates. The same is true for online ad campaigns that have an off-line impact or campaigns where there is a time lag between ad exposure and consumer action. Indeed, even something like search, which appears at first blush to be easily quantifiable, turns out to have an impact that extends in time far beyond the initial clicks on sponsored links. For example, in 2005 Comscore did a study of the impact of search within the consumer electronics category, which showed that only 2% of searches resulted in a sale within the same Internet session as when the search was conducted, but an astounding 25% of product searches resulted in a sale if one included offline sales and sales that occurred online but after the immediate exposure session. That is an astounding 12.5x understatement of search effectiveness if one were only to measure immediate action such as click through rate or same-session conversion. Only consumer panels offer the ability to accurately track the same consumer over time, in order to fully measure ROI.

Accurate Web site audience measurement is one of those early promises that has routinely been claimed, but just as routinely falsely delivered. Many Web site publishers look at their ‘internal data’ (which is almost always based on cookies) and automatically assume it is correct, without considering the impact of dynamics such as cookie deletion. Unfortunately, many in the media have made the same erroneous assumption. It is common to find sites that over-represent their audiences by a factor of 5 or more when citing server log data. Cookie deletion alone can account for 2.5x overstatement. Failing to exclude visitors from outside the US is another common oversight that can typically induce an overstatement between 1.5x and 4x. The compounding of these two factors alone can lead to an overstatement of between 3.75x and 10x. And this is before we take account of the inflationary impact of including ‘bot’ traffic, ‘push’ traffic and other factors.

I strongly believe that the industry does itself a disservice by continuing to publicly use flawed measurement metrics based on site server counts of cookies. The practice leads to a widespread impression of wild inaccuracies in online audience measurement, something that can only undermine the transparency and accountability of online media. Some even go as far as ‘longing’ for something ‘more accurate’ such as -- perish the thought -- TV ratings!

I hope our study helps educate people to some of the flaws in site-centric measurement. Don’t get me wrong. Site-centric measurement offers benefits in terms of tracking granularity and helping improving site design, so I am the last one to recommend against its use. What I do not recommend, is using it to publicly boast about the size of a site’s audience using metrics such as unique visitors that are unadjusted for the impact of factors such as cookie deletion. This can only lead to confusion, suspicion and, ultimately, loss of credibility for our industry.