Question

You have been talking to a student from the engineering program at UBC who has also taken up temporary residence at the North Pole in between active semesters of her studies. Because of her engineering background, the elves have given her the nickname “Casey Jones”. Casey scoffs at the α = .05 significance level that psychologists use in their hypothesis testing procedure. She tells you “That means you will, in the long run, make an error one time in twenty. If engineers had an error rate like that think of all the buildings and bridges and such that would be falling down! Why don’t you psychologists be more like engineers and set your error rate to something like one in a million instead of one in twenty?” Explain to Casey Jones why, in the context in which psychologists use it, an alpha level of one in twenty makes more sense than an alpha level of one in a million.

Answer 1

What does a zero p-value state? A zero p-value implies 100% confidence. What is the only statement that we can make with 100% confidence? That the expected value falls between negative infinity and positive infinity. Is that very useful? No, not really.

Okay, so big confidence intervals aren't very interesting, what about a very small confidence interval? If we choose a very small confidence interval, well, then how confident are we really that the true value falls within that small interval, defined by the expected value and a small standard deviation? Not very confident, it turns out.

So, these two limiting cases suggests that there is some optimum point that offers you the best bang for your buck - a reasonably small confidence interval at a reasonably high confidence.

p=0.05 corresponds to about a confidence interval of about two standard deviations - that means that we are 95% confident that the expected value falls within two standard deviations of the measured mean. That's pretty good! Consider the bell-graph, increasing our confidence (to 97%) even a tiny bit more increases the confidence interval significantly (to 3 standard deviations!), whereas decreasing the range quickly takes us away from 95% confidence. p=0.05 is kind of the sweet spot, if you will.