Radiohead released a statement today calling into question the validity of our recent report. Since we released our report on Monday, November 5, there has been a great deal of media coverage, and while much of the reporting has been excellent, there have been instances in both news media and blog postings that indicate not everyone understands how Comscore arrives at its market projections.
Comscore reports are derived from a representative sample of 2 million Internet users, who opt in to our panel and allow us to observe their actual online behavior, including e-commerce transactions. Because the data are based on passively observed consumer behavior, as opposed to polls or survey responses, there is no potential for recall error. When we observe an e-commerce transaction in our panel, the value we observe represents the actual price paid by that consumer.
As an affirmation of the validity and representivity of our panel, we regularly release quarterly U.S. e-commerce spending estimates, several weeks in advance of the U.S. Department of Commerce releasing its own figures, and during the past 7 years our figures have rarely deviated from the official Commerce numbers by more than a few percent.
For the Radiohead study, we observed the activity of nearly one thousand people who visited the “In Rainbows” site, a significant percentage of whom downloaded the album. We ultimately observed several hundred paid transactions, all of which ranged between $0-$20, representing a very robust sample for estimating the average price paid per transaction. It’s true that any sample has natural variability, so these numbers are, in fact, estimates. However, when you have a relatively large sample falling within a narrow range of values (i.e. there’s a small standard deviation), the margin of error in the estimate is minimized.
If you’re not yet convinced that sampling can produce an accurate estimate, think of it this way....
Let’s say you work in an office with 500 people and you want to find out how much the average person in the office spent on lunch today. You decide to randomly select 50 people from the office to get an estimate. Now you could ask them to tell you how much they spent on lunch, and you might get a reasonable estimate if they can remember. Some people might have forgotten what they spent, others may round to the nearest dollar, etc. But if you averaged the response of 50 people, you’d probably be in the ballpark.
Now what if, instead of asking them how much they spent on lunch, you asked for their lunch receipt. You would have data based on their actual observed behavior, and you would be able to calculate a much more accurate average. With the lunch receipts of 50 people in hand, how close of an estimate do you think it would produce? Because most people probably spend between $5-$10 on lunch (with a lunch of more than $15 being a rarity), you would get a very accurate estimate. Probably within pennies of the actual average.
This is effectively what we did with the Comscore study on Radiohead’s album sales. We observed the actual online spending behavior from a robust sample of hundreds of individuals in order to produce an accurate estimate. If we didn’t have a reasonable sample from which to extrapolate, we wouldn’t have released the data. But we did, and we’re confident in what the data showed.
I hope this helps clarify any confusion on the study.