Question

You are conducting a workshop on reliability for high school teachers and principals. One of the topics you are covering is reliability as it pertains to commercial published tests. You describe reliability evidence that test publishers typically present for their tests. You emphasize that, as consumers of tests, high school teachers and principals need to pay attention to that evidence and evaluate it. A principal, Mrs. Constantine, raises her hand to ask you a question: "You have made a really strong case for why we need to look carefully at the evidence for reliability that a test publisher includes in a technical manual. But what I don't understand is how high reliability needs to be for me to feel comfortable using scores from a commercial published test to make decisions about students."

How would you respond to Mrs. Constantine?

Write your one- or two-paragraph response to this question as if you were talking directly to Mrs. Constantine. Make certain that you are using teacher- and principal-friendly language as you respond to her (i.e., no measurement terminology unless you define terms).

In your response, identify two factors that high school teachers and principals need to consider when examining the degree of reliability presented in a technical manual for a commercial published test. Include at least two examples to illustrate the points you are making.

Answer 1

Mrs. Constantine, I completely understand your feeling regarding this matter. Reliability is one of important factor considered for a standardized test. Though the test publishers have established the reliability of this test, we have to ensure that this particular test is reliable without students. The test publishers have used a very different population to standardize the test. Our students are a different group. So it is better to establish reliability of this particular test in our students. Whether this particular test is providing us consistent results for our students is a big question for us. We have to do administer the test twice to our students and compare the results by data analysis. This gives you a correlation coefficient. If this particular value is closer to 1, then we can say there is a high correlation between the two scores. If the correlation comes out high, then we can confidently incorporate this test annually for the assessment. Suppose, we are administering a test on achievement motivation on students. The correlation coefficient of comes out 0.8. This shows that the test gives consistent results in our students. But if the correlation coefficient comes out as 0.1 it shows very low correlation between the test scores. See the same test can give out two different scores. We cannot say that the test is consistent here. Now, do you understand the concept of reliability in our case?