In: Statistics and Probability
What is the relationship between the simple two-sample t-test and the Bonferroni method of multiple comparisons?
The exact statement of your null hypothesis determines whether a Bonferroni correction applies. If you have a list of t-tests and a significant result for even one of those t-tests rejects the null-hypothesis, then Bonferroni correction (or similar).
Let's assume your hypothesis is "this instrument does not exhibit DIF", and you are going to test the hypothesis by looking at the statistical significance probabilities reported for each t-test in a list of t-tests. Then, by chance, we would expect 1 out of every 20 or so t-tests to report p≤.05. So, if there are more than 20 t-tests in the list, then p≤.05 for an individual t-test is a meaningless significance. In fact, if we don't see at least one p≤.05, we may be surprised!
The Bonferroni correction says, "if any of the t-tests in the list has p≤.05/(number of t-tests in the list), then the hypothesis is rejected".
What is important is the number of tests, not how many of them are reported to have p≤.05
If you wish to make a Bonferroni multiple-significance-test correction, compare the reported significance probability with your chosen significance level, e.g., .05, divided by the number of t-tests in the Table. According to Bonferroni, if you are testing the null hypothesis at the p≤.05 level: "There is no effect in this test." Then the most significant effect must be p≤.05 / (number of item DIF contrasts) for the null hypothesis of no-effect to be rejected.