Question

Embedded within the terminology and computations of statistics is the scientific nature of knowing. The scientific method, and its application in hypothesis testing, can never “prove” that something is “true.” Instead, a hypothesis test indicates whether or not the sample outcomes we have observed are consistent with a particular theory or claim that is being made about the phenomenon under consideration. In 300-500 words explain how this statistical basis of “truth” compares to the “absolute truth” we’d prefer. Include in your answer the role the selection of the alpha plays in this process.

Answer 1

First step is to discuss the theory of hypothesis testing, what it is and what it isn’t, as it’s fundamental to understanding the problem of providing evidence to support the null.

Hypothesis testing is confusing in part because the logical basis on which the concept rests is not usually described: it’s a proof by contradiction. For example, if you want to prove that a treatment has an effect, you start by assuming there are no treatment effects—this is the null hypothesis. You assume the null and use it to calculate a p-value (the probability of measuring a treatment effect at least as strong as what was observed, given that there are no treatment effects). A small p-value is a contradiction to the assumption that the null is true. “Proof”, here, is used loosely—it’s strong enough evidence to cast doubt on the null.

The p-value is based on the assumption that the null hypothesis is true. Trying to prove the null using a p-value is, therefore, trying to prove it’s true based on the assumption that it’s true. But we can’t prove the assumption that the null is true as we have already assumed it. The idea of a hypothesis test is to assume the null is true, then use that assumption to build a contradiction against it being true.

No conclusion can be drawn if you fail to build a contradiction. Another way to think of this is to remember that the p-value measures evidence against the null, not for it. And therefore lack of evidence to reject the null does not imply sufficient evidence to support it. Absence of evidence is not evidence of absence. Some would like to believe that the inability to reject the null suggests the null may be true . Failing to reject the null is a weak outcome, and that’s the point. It’s no better than failing to reject the innumerable models that were not tested. Although the null and alternative hypotheses represent a dichotomy (either one is true or the other), they underlie a parameter space. The alternative represents the complement of the space defined by the null, that is, the parameter space minus the null.