Question

1. How do you understand the concept of measurement, including measurement validity and reliability, in criminal justice research?

2. How do you view measurement issues in police auditing, particularly in performance audits?

3. What specific measurement criteria would you like to use, if you are hired to conduct an audit on police performance in crime clearance, that you believe would yield strong validity and reliability?

Answer 1

Answer 1. Measurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured.

Reliability

Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability).

Validity

Validity is the extent to which the scores from a measure represent the variable they are intended to. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever.

Answer 2. A significant proportion of today’s police organizations seem to remain narrowly focused on the same categories of indicators that have dominated the field for decades:

(a) Reductions in the number of serious crimes reported, most commonly presented as local comparisons against an immediately preceding time period.

(b) Clearance rates.

(c) Response times.

(d) Measures of enforcement productivity (e.g., numbers of arrests, citations or stop-and-frisk searches).

Performance Measurement

According to Neely et al (1995), performance measurement may be defined as the process of quantifying the efficiency or effectiveness of an action, while a performance measure is a metric used in the quantification process.

The words “quantifying” and “metrics” may suggest to some that performance measurement must include some type of counting. A sign hanging in Albert Einstein’s office refutes that suggestion, proposing that measurement is much more than just counting, particularly with respect to achieving qualitative objectives: “The problem is that not everything that counts can be counted, and not everything that can be counted, counts.” Whitaker et al (1982) discuss how best to measure, where he claims that observing is not measuring, as measuring requires the comparison of observation to a standard of measure.

In both the private and public sectors, performance measurement is considered a necessary component of good management. Experts in the field of management consulting have commented on the concept of performance measurement, with Peter Drucker, the well-known management guru stating: “If you can’t measure it, you can’t manage it.” Michael A. Lebowitz (2010: 1) expounded on that idea during a speech: “There's an old saying that if you don't know where you want to go, any road will take you there…this saying is mistaken. If you don’t know where you want to go, no road will take you there. In other words, you need an understanding of the goal.”

Nevertheless, it is evident that measuring police performance using traditional measures alone is not sufficient. To measure the police’s ability to arrest criminals only would be to completely disregard the other important work done by the police. The multi-dimensional nature of police’s work requires a multi-dimensional approach to the measurement of their performance. Researchers have thus resorted to indirect measures of police performance such as surveys, direct observations, and situational studies and independent testing. These are briefly summarized below.

Answer 3. Surveys can be a rich source of indicators that measure police performance. They may be administered by police jurisdictions, other state or provincial law enforcement authorities, academics, public opinion polling companies, or national institutions collecting statistical data. Surveys can be a very powerful tool to collect data that would supplement the traditional measures of police performance. The types of questions asked on surveys depend on which dimension of police work the researcher is attempting to measure. The wording of questions and their placement within the survey influence the answers of the respondents.

There are different types of surveys that researchers can employ in order to derive these indicators.

General community or national surveys usually ask questions about how the public views police performance, often resembling customer satisfaction surveys. Issues covered may include satisfaction with the work done by the police, the effectiveness with which police deal with social disorder and crime, whether individuals feel protected by the police, etc.
Contact or victimization surveys target individuals who were in contact with the police, or who were victims of crimes and dealt with the police following their incident. These kinds of surveys usually ask questions regarding individuals’ experiences dealing with law enforcement authorities.
Employee surveys are mostly internal surveys asking police employees about their work. They can be used to assess the levels of job satisfaction and the integrity of police employees.
Direct observations of social behavior or a certain phenomenon is a research method that is used widely in criminology. For example, the social disorder was observed by researchers in Chicago, where occurrences of the phenomenon were recorded on camera, coded, and analyzed (Sampson and Raudenbush, 1999). In police performance research, Mastrofski et al (1998) recorded and analyzed encounters between police officers and individuals to evaluate the behavior of police officers.

Situational studies/independent testing is yet another method that can be used to assess the performance of the police. In one example of this method, an American news network, ABC News, handed over 40 wallets that were supposedly found on the street to police officers in New York City and Los Angeles. All the wallets were returned with full contents in this police integrity test (ABC News, 2001, cited in Maguire, 2004). There were other studies conducted using this technique in the U.S., a notable example of which is by the Police Complaint Center, a Florida-based organization that conducts inquiries into police misconduct (Maguire, 2004). Although not widely practiced in academic research, probably due to ethical concerns and costs, the technique provides an objective alternative to other popular methods.