Question

88 Discuss the appropriateness of the frequentist practice of keeping the level of a test constant irrespective of the sample size.

89 What is the pre-testing problem? What are your views about this problem?

90 What is the post-selection problem? What are your views about this problem?

Answer 1

88. A (frequentist) hypothesis test, precisely, address the question of the probability of the observed data or something more extreme would be likely assuming the null hypothesis is true. This interpretation is indifferent to sample size. That interpretation is valid whether the sample is of size 5 or 1,000,000.

An important caveat is that the test is only relevant to sampling errors. Any errors of measurement, sampling problems,coverage, data entry errors, etc are outside of the scope of sampling error. As sample size increases, non-sampling errors become more influential as small departures can produce significant departures from the random sampling model. As a result, tests of significance become less useful.

This is in no way an indictment of significance testing. However, we need to be careful about our attributions. A result may be statistically significant. However, we need to be cautious about how we make attributions when sample size is large. Is that difference due to our hypothesized generating process vis a vis sampling error or is it the result of any of a number of possible non-sampling errors that could influence the test statistic (which the statistic does not account for)?

Another consideration with large samples is the practical significance of a result. A significant test might suggest (even if we can rule out non-sampling error) a difference that is trivial in a practical sense. Even if that result is unlikely given the sampling model, is it significant in the context of the problem? Given a large enough sample, a difference in a few dollars might be enough to produce a result that is statistically significant when comparing income among two groups. Is this important in any meaningful sense? Statistical significance is no replacement for good judgment and subject matter knowledge.

As an aside, the null is neither true nor false. It is a model. It is an assumption. We assume the null is true and assess our sample in terms of that assumption. If our sample would be unlikely given this assumption, we place more trust in our alternative. To question whether or not a null is ever true in practice is a misunderstanding of the logic of significance testing.

89. A pre-test is where a questionnaire is tested on a (statistically) small sample of respondents before a full-scale study, in order to identify any problems such as unclear wording or the questionnaire taking too long to administer. A pre-test can also be used to refer to an initial measurement (such as brand or advertising awareness) before an experimental treatment is administered and subsequent measurements are taken.

Pretests show what works with a particular audience. The slogan that seems persuasive to an adult may be confusing to a sixth-grader. A poster than captures the attention of teenagers in one community may alienate or seem boring to those in another.

Pretesting can prevent problems like these. It helps ensure that materials convey a clear and effective message about alcohol, tobacco, and other drugs to a program's target audience. Specifically, pretesting can help program managers:

• Select message concepts - styles, formats, spokesperson, and appeals (such as fear, humor, compassion)

• Guide creative work

• Fine tune wording and visual images

• Guide revisions (before spending time and money on the finished product).

Pretesting is valuable at several stages of message and material development. Some methods can be used in the early stages to test concepts or general issues and to spark ideas; other methods are more useful when materials are in close-to-final form.

Factors to consider in selecting a pretest method include, in addition to the stage of development of the materials, the kind of audience at which the materials are aimed (e.g., professional, rural), the sensitivity and complexity of the materials, and the resources available. The following overview summarizes the purposes, requirements, pros, and cons of six pretest methods.