Question

Can a Single Online Respondent Pool Offer a Truly Representative Sample?

Online research programs can often benefit by building samples from multiple respondent pools. Achieving a truly representative sample is a difficult process for many reasons. When drawing from a single source, even if researchers were to use various verification methods, demographic quotas, and other strategies to create a presumably representative sample, the selection methods them- selves create qualitative differences—or allow them to develop over time. The same is true of the parameters under which the online community or respondent pool was formed (subject matter mix, activities, interaction opportunities, etc.). Each online community content site is unique, and members and visitors choose to participate because of the individual experience their preferred site provides. As such, the differences between each site start to solidify as site members share more and more similar experiences and differences within the site’s community decrease. (Think, birds of a feather flock together.)

As such, researchers cannot safely assume that any given online respondent pool offers an accurate probability sample of the adult U.S. or Internet population. Consequently, both intrinsic (personality traits, values, locus of control, etc.) and extrinsic (panel tenure, survey participation rates, etc.) differences will contribute variations to response‐measure distribution across respondent pools. To control distribution of intrinsic characteristics in the sample while randomizing extrinsic characteristics as much as possible, researchers might need to use random selection from multiple respondent pools.

The GfK Research Center for Excellence in New York performed a study to see how the distribution of intrinsic and extrinsic individual differences varied between respondent pools. Respondents were drawn from five different online resource pools, each using a different method to obtain survey respondents. A latent class regression method separated the respondents into five underlying consumer classes according to their Internet‐usage driver profiles.

Researchers then tested which of the intrinsic characteristics tended to appear within the different classes. No variable appeared in more than three classes. Furthermore, the concentration of each class varied considerably across the five respondent pools from which samples were drawn.

Within the classes themselves, variations appeared in their demographic distributions. One of the five experienced a significant skew based on gender, and two other classes exhibited variable age concentrations, with one skewed toward younger respondents and the other toward older ones.

Overall, GfK’s study revealed numerous variations across different respondent resource pools. As their research continues, current findings suggest that researchers must be aware of these trends, especially in choosing their member acquisition and retention strategies and in determining which and how many respondent pools to draw from.

1. If one respondent pool is not sufficient, how many do you think you would have to draw from to get a truly representative sample? Why do you think that?
2. When creating a sample, how would you propose accounting for the types of extrinsic characteristics mentioned?

Answer 1

1. One respondent pool is not sufficient for getting a truly representative sample. A respondent pool is a group of individuals who will be participating in a survey. And a representative sample is simply a subset of a greater population the sample is made for.

The smaller the representative sample, the smaller the confidence levels and higher the margin of error.

A sample will always be a theoretical value dependent on a set confidence level and margin of error and the respondent pool shall be established mathematically according to the size of the polulation for which the study is conducted.

If x is the population, the representative sample shall be approximately x divided by 2 multiplied by a ratio of square of confidence levels to margin of error. When surveying, it is safe to consider the distribution as 50 % of the population.

2. Extrinsic characteristics are those characteristics that are not integral to the survey but they can also afect the survey. Factors like quality of people invigilating the survey, their knowledge, their term of involvement in the survey, survey participation rates (the ratio of number of individuals in the representative sample performing the survey to the number of people in represntative sample) etc.

There are many ways to account such factors. The quality of individuals performing survey can be regulated by audits and such methods. Tenure of the panel can be assessed and report be given to people managing the survey. For regulating participation rates, the survey can be made more interactive and fun. Limiting the time spent on surveys will ensure more people will participate in surveys. Random reward system can be employed. Survey can be conducted in all platforms like mobile phones, laptops etc. Adding a progress bar to the survey will be a good option as well.