Question

#1. Impact Evaluation

   In the move to online instruction due to COVID-19, Development University was concerned that its students may not have the technology they needed at home to keep up with their classes. In order to address this issue, at the start of the term, the University offered all students coupons for a 50% price discount on a new iPad. The hope was that the iPad would allow students to participate more fully in their online classes, and therefore allow them to learn more.

    Development University heard that you have learned how to do impact evaluation in your economics classes, so asked you to evaluate how well the iPads worked for their students. In particular, they wanted to know if the iPads caused student grades to improve. In looking at the data, you notice that although all students were offered the coupon, only half of them purchased an iPad through the program. Therefore, you decide to compare the average GPAs of students who purchased an iPad to those who didn't. You find that students who purchased an iPad had an average GPA of 3.4, and students who did not purchase an iPad had an average GPA of 3.1.

1) Due to your concerns, Development University decides to conduct a randomized control trial with incoming students in the fall. That is, incoming students are randomly assigned to either receive a coupon for a discounted iPad, or to not receive a coupon.

Will this research design allow you to estimate the true Average Treatment Effect of iPads on student performance? Why or why not?

2) In three sentences or less, briefly describe one concern you may still have with this randomized design.

Answer 1

1. In my opinion, this research design may not be appropriate enough to evaluate the effectiveness of the program. As much of a fact that availability of resource will have appositive impact on the students, the population that is being used to test this may need underlying factors to choose the samples. Students coming in fall for the course are mentally prepared for the course. Students generally tend to have devices like laptops, personal computers or smart phone already purchased with them before the course. There are some students however that may not have a device that can is computable with the online class format. When the research sample is selected on a random basis without any common basis, other factors are ignored that could have an impact on the GPA. A more organized format would be to use two groups with similar properties. Eg. Have one group of students with no other technology who are given the coupons and another group of students with no technology who do not get the coupon. Studying results from this group will give better more appropriate impact of the program.

2. This technique may not account for an equal representation of the actual population. The total number of students coming in fall may not be a good sample to test the impact on the entire population of the university. The number of students entering in fall may be less then usual due to the pandemic. Using these students for impact analysis may give a wrong response. Using a mix of old and new students would give better results.