Question

There are many controversies in the real life about the hypothesis testing. What do you think are the potential reasons for that? Can we find and share any example of misleading results of the hypothesis testing?

Answer 1

When a statistical hypothesis test produces significant results, there is always that chance that it is a false positive. In this context, a false positive occurs when you obtain a statistically significant P value, and you unknowingly reject a null hypothesis that is actually true. You conclude that an effect exists in the population when it actually does not exist.

The problem manily arise from

1. The alpha level (nearly always 0.1, 0.05, or 0.01) is arbitrary and without theoretical basis
2. In observational studies, the distribution of the test statistic under the null hypothesis is not known.
3. Presentation of only test statistics, degree of freedom and P-values limits the effectiveness . It is important to present parameter estimates and their precision

What should we focus on instead ?

Focus on effect size and its precision.

Do not rely on statistical hypothesis tests in the analysis of data from observational studies. With
strictly experimental data, use the usual methods (e.g., ANOVA and CANOVA) but focus on the
estimated treatment means and their precision, without an emphasis on the F and P values