Question

A school district developed an after-school math tutoring program for high school students. To assess the effectiveness of the program, struggling students were randomly selected into treatment and control groups. A pre-test was given to both groups before the start of the program. A post-test assessing the same skills was given after the end of the program. The study team determined the effectiveness of the program by comparing the average change in pre- and post-test scores between the two groups. During the course of the program, some students in both the treatment and control groups either dropped out of school or they moved away with their families. Could this attrition potentially distort the estimated effectiveness of the tutoring program? Defend your answer.

Answer 1

• attrition likely reduces the statistical power of the analyses of the available data
• attrition may compromise the external validity of the study, especially in those circumstances in which the likelihood of response is related to observed characteristics, such as race or place of residence. In those instances, the resulting parameter estimates may generalize only to a subset of the population of interest

Example, that all African American participants drop out of an evaluation. In that instance, the study's findings likely generalize only to the remaining racial and ethnic groups in the study.

• attrition is often linked to a range of characteristics in a complex manner. As a result, the population to which study findings generalize can be difficult to identify or describe.
• attrition also compromise the internal validity of a study. Particularly problematic are those instances in which the likelihood of response is related to the values of the variable for which values are only partially observed. This sort of attrition seems possible in many cases.

Example, an intervention targeting aggressive adolescents. If individuals who are incarcerated are less likely to participate in follow-up interviews, then aggression likely varies between those who do and do not participate.

Alternative ways to handle Attrition

1. Create a project identity.
2. Establish meaningful networks.
3. Consistently reward participants.
4. Develop and maintain a baseline
5. Inform participants about the schedule of follow-up interviews