Question

Dana uses the following parameters to determine that she needs a sample size of 140 participants for her study that will compare the means of two independent groups (t-test) using a one-tailed hypothesis:

            Effect size (d) = .5

            alpha = .05

            Power (1 - beta) = .90

Using the above information, answer each of the following questions.

a. If Dana keeps alpha and sample size the same, but desires an effect size of .80, what will happen to power? Will it increase or decrease? Explain.

b. If Dana keeps the desired effect size and sample size the same, but reduces alpha to .025, what will happen to power? Will it increase or decrease? Explain.

c. If Dana keeps power and sample size the same, but increases the desired effect size to .8, what will happen to alpha? Will it increase or decrease? Explain.

d. Dana decides she wants to increase the desired effect size to 2.0 and increase the power to .99, but keeps alpha the same. She does so to increase the likelihood that an effect will be found, and to make sure her results demonstrate a large enough effect. Also, when she conducts her a priori power analysis to determine her sample size, she is excited to see that she needs far fewer participants in her sample with those parameters (target n = 18). What, if any, are the problems with Dana’s strategy?

Answer 1

a.

The power will increase.

As the effect size(d) increases, the power of the test(​​​​​​) increases. Since effect size is increased from 0.5 to 0.8 keeping the sample size and alpha same, the power will increase.

b.

The power will decrease.

As the significance level(alpha) decreases, the power decreases. Since the significance level is reduced from 0.05 to 0.025, keeping the effect size and sample size same, the power will decrease.

c.

Alpha will decrease.

As the effect size increases, the significance level(alpha) decreases. Since the effect size is increased from 0.5 to 0.8, keeping the sample size and power same, the significance level will decrease.

d.

If we take two samples from the same population, there will always be a difference between them. The statistical significance is calculated as a p-value, i.e., the probability that a difference of at least the same size would have arisen by chance, even if there is actually no difference between the two populations. If p-value is less than the significance level, the difference is considered significant. This lower p-value could be achieved by increasing sample size or by increasing effect size (even though sample size is smaller). Thus, the problem with larger effect size is that it is only useful if there is really significant difference between the populations. Otherwise, it is not helpful and misleading. So, it is necessary to use sufficient sample size to find out if the difference is significant and effect size could be reported along with that. (so, larger effect size implies that we have already concluded the difference is significant).