In: Economics
Now that you have learned a bit about how Amazon uses A/B tests, what do you think are some of the risks and challenges associated with performing so many hypothesis tests? What would you do to mitigate those risks? Think about the results of the tests. What were the effects? Look at the size of the effects. Were they small or large? Why were the sample sizes so large?
Amazon uses A/B tests to analyse the information and thereby increase sales in the long run.
Some of the risks and challenges of performing so many hypothesis tests are that they are likely to create false results because there are bugs or the sample size is not the same. Varied hypothesis testing also makes the analysis inconclusive as companies get mixed results which reduces efficiency and focus on what they want to achieve. The company could make a change in terms of the product display or sale based on the hypothesis test which could reduce sales eventually if the test is not based on a large sample size and that there were mixed results.
I would increase the sample size, increase the focus on limited number of hypothesis tests so that the measures are implemented and could allot a certain time frame to check whether the change which has been done as per the results makes any difference in terms of sales and check it for several cycles, before making it permanent. One needs to do an educated guess in order to have a conducive effect.
Results could be biased depending on the size which could have large negative effects. The sample sizes were large because Amazon caters to a large audience, it cannot afford a discrepancy due to small sample size wherein there is no differentiation within customers. A large sample size increases efficiency of the test and helps increase accuracy in order to get a particular postive effect due to change in tactics.