In: Math

Describe the difference between statistical significance and effect
size. How is each used in describing the results of an
experiment?

**statistical significance:**

=>There is always some likelihood that the changes you observe in your participants’ knowledge, attitudes, and behaviors are due to chance rather than to the program

=>Testing for statistical significance helps you learn how likely it is that these changes occurred randomly and do not represent differences due to the program.

=>To learn whether the difference is statistically significant, you will have to compare the probability number you get from your test (the p-value) to the critical probability value you determined ahead of time (the alpha level).

=>If the p-value is less than the alpha value, you can conclude that the difference you observed is statistically significant.

**P-Value**: the probability that the results were
due to chance and not based on your program.

=>P-values range from 0 to 1.

=>The lower the p-value, the more likely it is that a difference occurred as a result of your program.

**Alpha (α) level**: the error rate that you are
willing to accept.

=>Alpha is often set at .05 or .01.

=>The alpha level is also known as the Type I error rate.

=>An alpha of .05 means that you are willing to accept that there is a 5% chance that your results are due to chance rather than to your program.

=>An alpha level of less than .05 is accepted in most social science fields as statistically significant, and this is the most common alpha level used in EE evaluations.

**effect size:**

=>When a difference is statistically significant, it does not necessarily mean that it is big, important, or helpful in decision-making.

=>It simply means you can be confident that there is a difference.

=>Let’s say, for example, that you evaluate the effect of an EE activity on student knowledge using pre and posttests.

=>The mean score on the pretest was 83 out of 100 while the mean score on the posttest was 84.

=>Although you find that the difference in scores is statistically significant (because of a large sample size), the difference is very slight, suggesting that the program did not lead to a meaningful increase in student knowledge.

=>To know if an observed difference is not only statistically significant but also important or meaningful, you will need to calculate its effect size.

=>Rather than reporting the difference in terms of, for example, the number of points earned on a test or the number of pounds of recycling collected, effect size is standardized.

=>In other words, all effect sizes are calculated on a common scale -- which allows you to compare the effectiveness of different programs on the same outcome.

=>There are different ways to calculate effect size depending
on the evaluation design you use. Generally, effect size is
calculated by taking the difference between the two groups (e.g.,
the mean of treatment group *minus* the mean of the control
group) and dividing it by the standard deviation of one of the
groups.

Describe how the following are related:
-Statistical significance and effect size
-Difference between means and effect size
-Effect size and variability
-Sample size and power of a statistical test
-Effect size and power of a statistical test

How are power, effect size, and sample size
related?
Distinguish between statistical significance and
practical significance.
What is the difference between a critical value and a
p-value?

What is the difference between practical and statistical
significance?
A. Statistical significance is associated with p values, but
practical significance is not.
B. Practical significance is associated with p values, but
statistical significance is not.
C. There is no difference.
D. Neither A nor B is true.

Explain the difference between statistical significance and
economic significance of OLS estimates?

Explain the difference between statistical and practical
significance.
Explain the difference between the null and alternative
hypotheses.
When should a one-tailed test be used? What are the
disadvantages to using a one-tailed test?
When should you use a two-tailed test?
Define a Type I error. In the behavioral sciences, what is the
likelihood of a Type I error?
Define a Type II error. In the behavioral sciences, what is the
likelihood of a Type II error?

1. What is the difference between practical significance and
statistical significance? Give an example of something that might
be statistically significant, but not practically significant.
2. What is a Type I error? Give an example.
3. What is a Type II error? Give an example.

Research Method
What is the difference between statistical significance and
clinical significance? Search the literature and cite at least one
or two examples that illustrate the differences between the
two.

What is the difference between measures of association and tests
of statistical significance? Give an example for each.

What is the difference between statistical and economic
significance? Give an example. (Your own example, NOT a pill for
cancer).

How do the significance level, power, effect size, and standard
deviation influence the sample size?

ADVERTISEMENT

ADVERTISEMENT

Latest Questions

- A firm has net sales of $5,500,000, Cost of Goods Sold $3,500,000, Depreciation Expense of $300,000,...
- Part A Determine the binding energy of radium-226 (22688Ra). Express your answer using five significant figures...
- 1. Bankers trust has bonds maturing in 27 yrs. the total principal that must be repaid...
- a) An eagle is flying horizontally at 4.4 m/s with a fish in its claws. It...
- analyze the financial impact of johnson and johnson settlement on the stock price. As an analyst,...
- mutual funds and choose a fund in which you might like to invest. Identify this fund...
- Consider the following two mutually exclusive projects: Year Cash Flow (A) Cash Flow (B) 0...

ADVERTISEMENT