Question

An important part of using statistics is being able to explain your results to decision makers. Imagine that you have conducted a two-sample test and determined that the difference was not statistically significant.   While one mean was 4.3 and the other was 3.9, the p level for the t test was p=.07. Your management team says, “Well, the difference may not be statistically significant, but the difference is there! Discuss how you would respond and how you would explain the purpose of the t test and significance in this case.

Answer 1

In 2 Samples t test:

1 = Mean of Sample 1 = 4.3

2 = Mean of Sample 2 = 3.9

The difference:

= 4.3 - 3.9 = 0.4

can be attributed to two reasons as follows:

Reason 1: Difference due to statistical nature of the problem,i.e., the difference is purely due to random variation and not due to any assignable cause

Reason 2: Difference due to an assignable cause.

In this case, it is noted from the p -value = 0.07, the difference is due to random errors only and there are no assignable causes. The difference is statistically insignificant.

So, answer to the management is:
Difference is there. This is accepted. In fact, in all experiments, difference will always be there. But, the question is: whether the difference is beyond tolerance. If it is beyond statistical significance, it is attributed to external cause and we should not neglect the difference. Otherwise, we can conclude that there is no significant difference.