Question

Regardless of the outcome in the statistical test, either Jason or Andrew's claim would be supported. This could hurt their friendship. As a more seasoned statistician, highlight three possible concerns in business or statistical aspects regarding the approach used here(Confidence Intervals and Hypothesis testing) so that the tension between Jason and Andrew could be eased. If you have a free hand on the project starting from sampling to anlysis, how would you conduct the project analysis and what recommendations would you make for the problem in question?

Answer 1

A hypothesis test allows us quantify the probability that our sample mean is unusual.

To determine whether the probability is small, we will compare it to the preset level of significance, which is the probability of Type I error. Recall that Type I error is the more serious error - to reject.

The significance level defines the distance the sample mean must be from the null hypothesis to be considered statistically significant.
The confidence level defines the distance for how close the confidence limits are to sample mean.

In statistical analyses, there tends to be a greater focus on P values and simply detecting a significant effect or difference. However, a statistically significant effect is not necessarily meaningful in the real world. For instance, the effect might be too small to be of any practical value.

It’s important to pay attention to the both the magnitude and the precision of the estimated effect. That’s why I'm rather fond of confidence intervals. They allow you to assess these important characteristics along with the statistical significance. You'd like to see a narrow confidence interval where the entire range represents an effect that is meaningful in the real world.