The Problem With The Pew Study

December 31, 2010 3:29 am

“65% of internet users have paid for online content” – you will see this headline all across the internet in this coming week. Thanks to a survey by the Pew Research Center, these numbers will be interpreted in multiple ways and mostly used to convince stakeholders of all categories that people have finally started paying for things on the Internet. The full report is here.

So whats the issue with the study? The one number from the report that will NOT be quoted is this: 1003. That number is the total number of people that participated in the survey.

From the report:

The PSRAI October 2010 Omnibus Week 4 obtained telephone interviews with a nationally representative sample of 1,003 adults living in the continental United States. Telephone interviews were conducted by landline (672) and cell phone (331, including 134 without a landline phone). The survey was conducted by Princeton Survey Research Associates International (PSRAI). Interviews were done in English by Princeton Data Source from October 28?November 1, 2010. Statistical results are weighted to correct known demographic discrepancies. The margin of sampling error for the complete set of weighted data is ±3.7 percentage points.

Definition of sampling error from Wikipedia:

The likely size of the sampling error can generally be controlled by taking a large enough random sample from the population,[2] although the cost of doing this may be prohibitive; see sample size and statistical power for more detail. If the observations are collected from a random sample, statistical theory provides probabilistic estimates of the likely size of the sampling error for a particular statistic or estimator. These are often expressed in terms of its standard error.

So here is what is missing from the Pew study: Where are the probabilistic estimates of the likely size of the sampling error? The report quotes some percentage points – what are the calculations that have gone into it?

And for everyone claiming that most internet users now pay for online content – this is classic Narrative Fallacy. Yes there has been a survey and there are results. Yes there has been an uptick in online content purchase activity. But connecting these is Narrative Fallacy – our need to fit a story or pattern to a series of connected or disconnected facts.

For Interesting Statistics Everyday, Find Statspotting on Facebook and Follow Statspotting on Twitter

Leave a Reply