Question

As you just saw, we used a Tukey post hoc test to look at the differences between the sloppy, dressy, and casual conditions. So why not just use three t-Tests (one comparing sloppy to dressy, one comparing sloppy to casual, and one comparing dressy to casual)? For this Pause-Problem, tell me why you wouldn’t want to run multiple t-Tests.

Answer 1

The reason is that if we perform multiple t test 2 things happen

1. type 1 error compunds
2. and we only use a subset of data

assume with α= 0.05, if the null hypothesis is true then the probability of not obtaining a significant result is 1 – 0.05 = 0.95.

Multiply 0.95 by the number of tests to calculate the probability of not obtaining one or more significant results across all tests. For two tests, the probability of not obtaining one or more significant results is 0.95 × 0.95 = 0.9025.

Subtract that result from 1.00 to calculate the probability of making at least one type I error with multiple tests: 1 – 0.9025 = 0.0975.

lets say that
You are comparing 4 groups (A, B, C, D). You compare these six pairs (α= 0.05 for each): A vs B, B vs C, C vs D, A vs C, A vs D, and B vs D.
· Using the formula , the probability of not obtaining a significant result is 1 – (1 – 0.05)^6 = 0.265, which means your chances of incorrectly rejecting the null hypothesis (a type I error) is about 1 in 4 instead of 1 in 20!!