John Locke Update / Research Brief

The Debate Over Risk Assessment Tools in Pretrial Detention

posted on in Law & Regulation, Legal Update
Featured Image

When it comes to public policy, extreme, ideologically-driven posturing has become the norm in many parts of the country. Here in North Carolina, however, people tend to take a more moderate, pragmatic approach. Both regional tendencies were recently on display in a debate over the use of so-called “risk assessment tools,” i.e., computerized systems that use actuarial data to help magistrates and judges decide what should happen to criminal defendants prior to trial.

In North Carolina, as in most states, when magistrates and judges make that decision, they are supposed to choose the least restrictive option necessary to ensure that the defendant ultimately appears at trial and does not pose a danger to the public in the interim. Making the right call is difficult, however, and there’s a lot at stake. The decision-maker must consider, not just all the relevant facts about each specific defendant but a great many other factors as well. One of those factors is that, in order for the defendant to have been charged, there must at least be probable cause to believe that he or she has already broken the law. That being the case, it’s not unreasonable to suspect that, if the defendant is released, he or she may abscond or commit another crime prior to trial. On the other hand, under both the U.S. and the North Carolina constitutions, each defendant is presumed to be innocent until proven guilty and may not be deprived of liberty without due process of law. Further complicating matters, pretrial detention imposes significant costs, not just on taxpayers, but also on criminal defendants and their families, and it makes economic and moral sense to minimize those costs as much as possible.

Critics of the current system claim that magistrates and judges systematically err on the side of caution, and, as a result, too many defendants are unnecessarily detained prior to trial. The critics may well be right. Making the right pretrial determination in every case would be hard under the best of circumstances, but it’s made even harder by the fact that, in most jurisdictions, it must be done for large numbers of defendants in quick succession.

To improve matters, a growing number of magistrates and judges, including some in North Carolina, have been using computerized risk assessment tools to help them make objective decisions based on data rather than intuition, and the results so far have generally been good. Nevertheless, last month more than two dozen academics, most of them most of them from Harvard and MIT, released a joint statement urging judicial decision-makers to abandon the use of such tools. Here’s the summary that appears at the beginning of the statement:

Actuarial pretrial risk assessments suffer from serious technical flaws that undermine their accuracy, validity, and effectiveness. They do not accurately measure the risks that judges are required by law to consider. When predicting flight and danger, many tools use inexact and overly broad definitions of those risks. When predicting violence, no tool available today can adequately distinguish one person’s risk of violence from another. Misleading risk labels hide the uncertainty of these high-stakes predictions and can lead judges to overestimate the risk and prevalence of pretrial violence. To generate predictions, risk assessments rely on deeply flawed data, such as historical records of arrests, charges, convictions, and sentences. This data is neither a reliable nor a neutral measure of underlying criminal activity. Decades of research have shown that, for the same conduct, African-American and Latinx people are more likely to be arrested, prosecuted, convicted and sentenced to harsher punishments than their white counterparts. Risk assessments that incorporate this distorted data will produce distorted results. These problems cannot be resolved with technical fixes. We strongly recommend turning to other reforms.

On the same day, three of the joint statement’s authors published an opinion piece in a New York Times in which they characterized risk assessments as “a case study of how a real-world ‘Minority Report’ doesn’t work.”

Exhibiting exactly the kind of reasonable common sense I’ve come to expect from North Carolinians, three local academics—Sarah Desmarais, who’s a professor at NCSU, and Brandon Garrett and Cynthia Rudin, who are professors at Duke—responded with a joint statement of their own. After acknowledging that the statement attacking risk assessment was “signed by a number of academics whom we respect,” Desmarais, Garrett, and Rudin went on to refute the statement point by point. Among other things, they said:

The author’s claim that risk assessments do not accurately measure pretrial risks misses the fact that these tools estimate relative risk, not absolute risk. Uncertainty is wide when predicting crime over short time intervals, but smaller when predicting over larger intervals. The latter risks are directly related to the former, allowing tools to more confidently assess which individuals are higher risk than others. That is why risk assessment tools are evaluated by how well they can rank individuals from high to low risk. While there are technical challenges, it is extreme to claim that no remedy exists, and to insist that we make decisions without using data and statistics. In fact, solutions actually do exist.

The authors also criticize the use of criminal history data in pretrial risk assessment. … However, that is exactly the kind of information that judges consider when they don’t have the benefit of these tools. One would be hard-pressed to find a set of bail and sentencing guidelines that does not include criminal history as a major component in a person’s pretrial release determination or sentence.

Regarding the claim of racial bias in risk assessment tools, we would point out that a defendant’s race is quite evident in the courtroom, making the defendant subject to any racial biases a judge might hold. …

Following these and other specific refutations, they concluded:

A judge often sets cash bail, which punishes the poor, and may allow wealthy criminal defendants to walk free. Judges and magistrates make these calls very, very quickly, often relying almost exclusively upon criminal and charges records, without meaningful input from defense lawyers. The result fills our jails in ways that can harm public safety.

To call risk assessment fundamentally flawed suggests that we should abandon reforms and keep things the way they are. Instead, we need to give judges better information. No human being is an expert predictor. Relying on empirical data is far superior to going with one’s gut, if it is the right data, carefully analyzed, and presented in such a way as to minimize bias. In fact, statistical tools can be specially designed to help reduce the biases that are — obviously — inherent in the data. …

The critics of risk assessment themselves require a dissenting opinion: a minority report. While risk assessment tools may not eliminate racial, ethnic or other biases, there is no evidence that they exacerbate them either. Risk assessment tools and the promise they hold to improve on judges’ and magistrates’ current decision-making processes should not be dismissed simply because they aren’t yet perfect.

This is clearly the right approach, not just to risk assessment, but to criminal justice reform in general. By taking the kind of common-sensical, fact-driven approach advocated by Desmarais, Garrett, and Rudin, a bipartisan coalition of criminal justice reform advocates have accomplished a great deal over the last fifteen years, not just here in North Carolina, and across the country. Extreme, ideologically driven posturing could destroy that coalition and make future criminal justice reform much more difficult. And if such posturing inspires ill-considered reforms that cause crime rates to rise, it could lead to a backlash that would threaten all the criminal justice reform that has already occurred. To prevent such outcomes, the ideologues in other parts of the country would do well to follow the example of their more sensible peers in North Carolina.

Jon Guze is Senior Fellow in Legal Studies at the John Locke Foundation. Before joining the John Locke Foundation, Jon practiced law in Durham, North Carolina for over 20 years. He received a J.D., with honors, from Duke Law School… ...

Donate Today

About John Locke Foundation

We are North Carolina’s Most Trusted and Influential Source of Common Sense. The John Locke Foundation was created in 1990 as an independent, nonprofit think tank that would work “for truth, for freedom, and for the future of North Carolina.” The Foundation is named for John Locke (1632-1704), an English philosopher whose writings inspired Thomas Jefferson and the other Founders.

The John Locke Foundation is a 501(c)(3) research institute and is funded solely from voluntary contributions from individuals, corporations, and charitable foundations.