Question

Blake has developed a new measure of personality to assess what personality type people have. It is called the “Blockbuster Personality Type” measure. The measure includes images from the top ten most popular Hollywood movies in the last ten years and asks respondents to describe what is happening in the scenes. Responses are scored in terms of how consistent they are with the original movies, and how much respondents show empathy for the different characters in the scenes. As a result of this scoring, respondents are classified as either being either “active heroes”, “passive victims”, or “conniving villains”, reflecting the hidden desires and goals that they were born with.

Blake wants to encourage people to use the test for selection and recruitment in organisations around the world, so he collects information about the test’s reliability and validity. He asks his first-year psychology class to take the test at the start of semester and at the end of semester and compares their scores for a measure of reliability. To test validity, he uses scores on the test to predict how well his students do in their university studies.

Question: What are some of the issues with what Blake has done? How could he have designed and evaluated his test in a better way? For each issue you identify, explain why it is an issue and how it could be improved.

Answer 1

There are several issues with Blake's test:

1. It appears to have poor construct validity. Although the test claims to measure an individual's personality type, it is also assessing the respondent's knowledge about the movie scene. This factor should be eliminated and Blake should only focus on the extent to which respondents show empathy for the characters.

2. The responses are open-ended and this can lead to huge amounts of subjectivity when it comes to scoring. Blake should attempt to collect close-ended responses to reduce this possibility.

3. This test is likely to be very culturally biased as all cultures cannot be expected to have knowledge about Hollywood blockbusters. It would be better if Blake included a description of the scenes for test takers who may not have a background knowledge of the movies.

4. Blake's measure of the test validity is also faulty. Since it is a personality test, he should not be using it to predict students' performance on university studies.