Question

In a hypothesis test, the significance level α (alpha) represents the threshold for how unlikely sample data has to be, assuming the null hypothesis, in order for us to reject the null hypothesis. Often alpha is taken to be 0.05, or 1/20, which means that even if the null hypothesis is true, there is a 1/20 chance that we will reject it because our sample data happens to have a P-value less than 0.05.

We want to study the heights of students. Our null hypothesis H_0 is that the average height is 67 inches, and the alternate hypothesis is that the average height is not 67 inches. We use significance level 0.05.

A) Suppose we run a hypothesis test. In the test, the data had a P-value less than 0.05. What should the conclusion of the test be?

B) Suppose we run 20 hypothesis tests on this, each with a different sample. In nineteen of these tests, our sample data had a P-value greater than 0.05. In one test, the data had a P-value less than 0.05. Does this evidence, as a whole, support the hypothesis that the average height is 67 inches? Explain your thinking.

Answer 1

(A) When the p value is less than the alpha level, then reject the null hypothesis and we conclude that the claim is supported by the sufficient evidences.

So, if the p value is less than 0.05, then we will reject the null hypothesis and we will conclude that we have enough evidences to state that the average height is not 67 inches

(B) It is given that the p value in 19 tests is greater than 0.05, means the results were insignificant in 19 tests. Only one test out of 20 had shown p value less than 0.05, means we have 5% inaccurate data (5% out of 100% is same as that of 1 out of 20)

Since we are using 0.05 significance level or 5% significance level, this means we can have a probability error of 5% of making incorrect decision.

So, it is normal to have 1 incorrect p value when having 20 hypothesis tests. Yes, evidence as a whole support the hypothesis that the average is 67 inches because we can have 5% error in hypothesis tesing when using 0.05 significance level.