Question

**Experimental Research Design**

Explain the general advantages and disadvantages of within-subjects designs compared to between-subjects designs. Give an example of a study using one of the designs and explain why that design is best for the proposed study.  

Answer 1

When you want to compare several user interfaces in a single study, there are two ways of assigning your test participants to these multiple conditions:

Between-subjects (or between-groups) study design: different people test each condition, so that each person is only exposed to a single user interface.
Within-subjects (or repeated-measures) study design: the same person tests all the conditions (i.e., all the user interfaces).
For example, if we wanted to compare two car-rental sites A and B by looking at how participants book cars on each site, our study could be designed in two different ways, both perfectly legitimate:
Between-subjects: Each participant could test a single car-rental site and book a car only on that site.
Within-subjects: Each participant could test both car-rental sites and book a car on each.

Any type of user research that involves more than a single test condition has to determine whether to be between-subjects or within-subjects. However, the distinction is particularly important for quantitative studies.

Comparision:
Between-subjects minimizes the learning and transfer across conditions. After a person has completed a series of tasks on a car-rental site, she is more knowledgeable about the domain than she was before. For example, she may now know that car-rental sites charge an extra fee for drivers under 21, or what a collision-damage waiver is. That knowledge will likely help her become more efficient on a second car-rental site, even though that second site may be very different from the first.
With between-subject design, this transfer of knowledge is not an issue — participants are never exposed to several levels of the same independent variable.

Between-subjects studies have shorter sessions than within-subject ones. A participant who tests a single car-rental site will have a shorter session than one who tests two. Shorter sessions are less tiring (or boring) for users, and can also be more appropriate for remote unmoderated testing (especially since tools like UserZoom usually require a fairly short session length).
Between-subject experiments are easier to set up, especially when you have multiple independent variables. When the study is within-subjects, you will have to use randomization of your stimuli to make sure that there are no order effects. For example, in our car-rental study, we need to make sure that participants don’t always start with site A and then move on to site B. The order of the sites needs to be random for each participant. This is easy with just two sites: randomly assign 50% of users to start with each site. But as you increase the number of independent variables and of levels for an independent variable, randomization becomes more difficult to implement within some of the existing platforms for quantitative usability testing.
Within-subject designs require fewer participants and are cheaper to run. To detect a statistically significant difference between two conditions, you’ll often need a fair number of a data points (often above 30) in each condition. If you have a within-subject design, each participant will provide a data point for each level of the independent variable. For our car-rental study, 30 participants will provide data points for both sites. But if the study is between-subjects you will need twice as many to get the same number of data points. That means twice the cost.
Within-subjects design minimize the random noise. Perhaps the most important advantage of within-subject designs is that they make it less likely that a real difference that exists between your conditions will stay undetected or be covered by random noise.
Individual participants bring in to the test their own history, background knowledge, and context. One may be tired after a long night of partying, another one may be bored, yet another one may have received a great news just before the study and be happy. If the same participant interacts with all levels of a variable, she will affect them in the same way. The happy person will be happy on both sites, the tired one will be tired on both. But if the study is between-subjects, the happy participant will only interact with one site and may affect the final results. You’ll have to make sure you get a similar happy participant in the other group to counteract her effects.
In practice, researchers won’t be able to assess such differences between participants — although they may match the gender, the experience, and the age across groups, it will be difficult to predict or detect other factors specific to each participant.