We have often made the case for government transparency and measurement to improve government performance and outcomes. Consultants Katherine Barrett and Richard Greene provide examples of how bad incentives and bad motives can lead to bad reporting of numbers.
- Altering scores: The infamous example of 11 Atlanta teachers who were convicted for fudging standardized test scores “under pressure to meet certain scores or…[risk] termination.”
- Throwing away bad results: a New York City staff person assigned to help with customer service training “threw away the results of the people who had the lowest scores”
- Inflating numbers: “If there’s an oil spill and fish die, you send a biologist out to count dead critters. They get a [certain] amount for each dead critter [they find], and they send you the bill.”
- Changing the definitions: In 2017, New York City “changed the definition of critical incidents [at homeless shelters] to involve firearms possession, but not other weapons commonly found in shelters, such as shivs, steak knives, and locks in socks.”
- Stopping the clock: UNC School of Government’s David Ammons notes that the clock for emergency response times does not start when a 911 call comes in, but only when the emergency unit is dispatched.
Sometimes, though, data can look worse because people are looking. Phoenix is one of the few cities with a Bias Crimes Unit and reports every time an officer suspects bias. “As a result, more cases are identified, investigated and reported,” which makes the city look worse on cross-city comparisons.