Question

Candle-Line is an online retailer that does frequent A/B testing. The process of running a test is as follows: The first step is designing the test; this involves the web design team. The test then waits for its turn to be run. Next, the test is run and data is collected. Once the test is completed, the data must wait for an available statistician. When a statistician is available, she performs the analysis and generates a report. Tests originate in two ways. They may arise from outside the web design team (an “external” test) or from the web design team (an “internal” test). Internal tests do not require the Design phase and go immediately to Wait to Run. 10% of tests are internal. Design for an external test takes on average 4 days. On average, a test (whether internal or external) spends 2 days in Wait to Run. The length of the experiment varies: 20% of tests run for 14 days while the remaining run for 7 days. On average, there are 84 tests running. At the end of the experiment, 20% of tests are deemed redundant and thus are abandoned without any processing. The remaining tests wait for a statistician; there are on average 16 tests waiting for a statistician. Doing the analysis and generating the report takes 2 days on average.

a) On average, how many reports per day are the statisticians generating?

On average the statisticians generate ___________ reports per day.

b) What is the average flow time for a test (regardless of whether it is internal or external and whether its data is abandoned or analyzed)?

The average flow time is ___________ days.

c) Designing an external test costs $100. Candle-Line spends $50 dollar/day for each test that is running. Generating a report cost $500. What is their weekly spending on A/B testing (assume 7 days per week)?

Their weekly spending on A/B testing is $__________________ per week.

Answer 1

Answer: a

On average the statisticians generate 0.5 reports per day.

Statistician generates one report in 2 days, thus the report generated per day = 1/ 2 = 0.5 reports per day.

Answer b:

The average flow time is 8.4 days.

20% tests are run for 14 days

80% tests are run for 7 days

If we take a example of 20 tests with lead time of 14 days and 80 tests with lead time of 7 days.

Average = (20*14) + (7*80) / 100 = 840 / 100 = 8.4 days / test.

Answer c:

• Design cost of external test costs = $100.
• Candle-Line spends $50 dollar/day for each test that is running.
• Generating a report cost $500.

We need to calculate total tests at each stage for a weeks time

Weekly Design step = 1 external test takes 4 days = 8 days required for 2 test to be designed. Thus 1.75 tests can be designed in 7 days

Design cost = No. of design tests * design test unit cost = 1.75* 100 = $175

Weekly Run Test = One tests runs either for 7 days or 14 days base don their nature. Unit rate per test /day = 50. Total avaregate tests are running = 84 nos.

Run Test Cost = Days * Test rate per day* no. of tests = 7*50 * 84 = $ 29400

Weekly Report Cost = Total report * Report unit cost = 3.5 Report * 500 = $1750

Their weekly spending on A/B testing is $_31325___ per week

= Weekly Design cost + Weekly Run cost + Weekly report cost

= 175 + 29400 +1750 = $ 31325

Refer the sheet for more details as below

Type of Tests	Internal test	10%	No design	Wait for test Run	Test Run & data Collection	Test Outcome - 20% fail Rate, 80% Pass rate	Test Data waits for analysis	Statistical Analysis and Report generation
	External test	90%	Test Design
Lead Time			4 days	2 days	20% for 14 days 80% for 7 days			2 days/report
Average Tests					84 test running		16 tests