Question

A city department plan to administer a survey to a sample of citizens to asses program quality. Department managers intend to measure citizens satisfaction with service and want to make sure that the survey instrument contains reliable and operationally valid measures.

a. Present two strategies that you can use to determine that the measure of program quality are reliable.

b. Present two strategies that you can use as evidence that the measures of program quality are operationally valid.

Answer 1

1. Two strategies that can be used to determine that the measure of program quality are reliable:
A. Test-retest reliability: Have a set of participants answer a set of questions (or perform a set of tasks). Later (by at least a few days, typically), have them answer the same questions again. When we correlate the two sets of measures, look for very high correlations (r > 0.7) to establish retest reliability.
B. Internal consistency reliability: Using a statistical software, the Cronbach's alpha can be calculated. The more items we have, the more internally reliable the instrument. We should look for a Cronbach’s alpha of 0.9 and above.

2. Two strategies that you can use as evidence that the measures of program quality are operationally valid:
A. Convergent validity: Have the participants take another similar survey that also assesses program quality. Theoretically, we should expect that the correlations between the two measures (our scale and the scale similar to our scale) are positive and high.
B. Divergent validity: Here, we need to have the participants take a survey in which items are assessing complaints or dissatisfaction with the program quality. We should expect that the correlation between our scale and the scale that measures the construct dissimilar to ours to be negatively related.