Question

Assume that a researcher is intending to compare the means of three different treatment groups. The researcher is concerned about the fact that multiple comparisons will be made during the analysis phase. A) What is the probability that the researcher will reject at least one true hypothesis in his study? B) Assume the researcher is going to compare the means of four different treatment groups. What is the probability that the researcher will reject at least one true hypothesis in his study? C) What happens as you increase the number of comparisons to be made? D) How do researchers overcome this phenomenon?

Answer 1

Assuming significance level of 0.05,  probability that the researcher will reject one true hypothesis in his study = 0.05

Probability that the researcher will not reject one true hypothesis in his study = 1 - 0.05 = 0.95

A)

To compare the means of three different treatment groups, we need to perform ³C₂ = 3 tests.

Probability that the researcher will reject at least one true hypothesis in his study = 1 - Probability that the researcher will not reject any true hypothesis in his study = 1 - 0.95³ = 0.142625

B)

To compare the means of four different treatment groups, we need to perform ⁴C₂ = 6 tests.

Probability that the researcher will reject at least one true hypothesis in his study = 1 - Probability that the researcher will not reject any true hypothesis in his study = 1 - 0.95⁶ = 0.2649081

C)

We see that the probability that the researcher will reject at least one true hypothesis in his study increases with the  increase in the number of comparisons to be made.

D)

To overcome this phenomenon, use the statistical methods like ANOVA test to compare the the means of more than two different treatment groups.