Question

Conceptually, what are we doing when we test for statistical significance (such as in a z-test or t-test)? Where does the commonly used 95% confidence level come from? What is an effect size and what additional information does it provide about a finding?

Answer 1

Engineering problems require to decide whether to accept or reject a claim or statement about a parameter. This statement is called hypothesis and the procedure to solve the problem is called hypothesis testing. After the data has been collected, statistical inference is carried out to test if evidence or claim is in favour or some conclusion is drawn. This method of statistical inference is called tests of significance (such as z test and t test).

Basically we have 2 types of hypothesis –

1. Null hypothesis ( H₀ ): Claim that is believed to be true but not proved yet.

2. Alternate Hypothesis ( H_A ): Statement of what a statistical hypothesis test is trying to prove. It’s the opposite of the null hypothesis.

While testing for statistical significance the claims or statements are tested for population parameters like mean, variance and population proportion.

An interval estimate for a population parameter is called confidence interval. We generally use the 95% C.I so that we have high confidence it contains the unknown population parameter. Another reason for using 95% C.I is that it leads to higher precision which means it will most likely contain the true value for the unknown population parameter. The critical values for the 95% C.I can be obtained from the standard tables or can be obtained from STATKEY.

Effect size the measure of strength of a phenomenon. It explains the difference in the strength of relationship between two variables. For example, if we have a data on the weights of 50 boys and 50 girls, and obviously boys are heavier compared to girls, so the difference between the weights of the boys and girls is the effect size. Greater the effect size, greater is the weight difference between them.

Additional information provided by effect size –

Effect size provides a more scientific approach for determining the relationship between two variables compared to p value, and it can be used to compare the results of studies in different scenarios.