Question

Explain why a difference-in-means analysis risks presenting a biased estimate of the effect of the policy.

Answer 1

Difference-in-mean measures the absolute difference between means of two groups/sample in a trial. It gives us an understanding of the variation in the average of experimental and control group. It is important to highlight here that the difference measured is an “absolute difference” i.e the positive or negative sign don’t matter.
Difference-in-means analysis runs the risk of a biased estimate if the two population under consideration do not have the same variance or if the population are not normally distributed. Again, bias can also arise if each value is not sampled independently. If multiple values are factored-in from single source, that is not considered independent and the result will not be free from bias.
Bias in difference-0n-mean can have a detrimental effect in policy decision making. For example, if we are comparing the correlation of an age group with a particular disease, then the bias that crop up in conducting the activity can jeopardize the findings and subsequent decision making. For better correlation of cause and effect, it is important that the difference-in-mean measurement are performed accurately.