A series of recent scandals involving forensic evidence in North Carolina have placed hundreds of convictions in doubt and undermined public and judicial confidence in forensic science. To address the problem, procedural and personnel changes have been made at North Carolina’s State Bureau of Investigation (SBI) crime lab. Three recent publications, however, suggest that the underlying problem didn’t originate at SBI and won’t be solved simply by changing some of its procedures and firing some of its employees. There’s a national crisis in forensic science, and it’s far from over.

Earlier this month, the Sandra Day O’Connor College of Law at Arizona State University published Reforming Criminal Justice, a comprehensive four-volume report that was funded by the Charles Koch Foundation and prepared by the Academy for Justice. Among its 57 chapters is one called “Forensic Evidence” written by New York University Law Professor Erin Murphy.

Professor Murphy begins the chapter by observing that:

The scientific integrity of forensic evidence depends on … three safeguards: its methodological validity, its statistical validity, and its execution in a particular case. Unfortunately, … many familiar forensic techniques have long been admitted as evidence in criminal cases without meeting all – and often times any – of these foundational requirements.      

She then goes on to review how well, or poorly, the foundational requirements are met by various commonly used techniques, and her findings are damning. While she finds that some techniques, such as DNA and drug testing, are generally reliable – at least when properly executed – she casts serious doubt on the reliability of many others. For example:

Fingerprint [identification] … susceptible to analyst bias, and has a high false-positive rate. 

Firearm/toolmark identification … currently falls short of the criteria for foundational validity.

Bite-mark evidence … lacks any scientific foundation.

Impression/pattern analysis of shoes, tires, etc. … unsupported by statistical surveys, and may mislead jurors.

Fiber analysis … no set standards.

Hair analysis … a recent review by the FBI found that 90% of cases involving hair analysis contained erroneous statements.

It’s not all bad news. Professor Murphy offers many suggestions about how matters might be improved, and she praises several reform measures that have already been implemented, including the oversight commission established by Texas in 2005, and the Forensic Resource Counsel created by North Carolina’s Office of Indigent Defense Services in 2010. Nevertheless, her overall finding is that the forensic science system is badly broken and in need of repair.

The second recent publication dealing with the forensic science crisis is an opinion piece that appeared in the Proceedings of National Academy of Sciences (PNAS) on April 12th. In “A call of more science in forensic science,” Susan Bell, who teaches Forensic and Investigative Science at West Virginia University, and five co-authors from a variety of disciplines including physics, chemistry, biology, and public health wrote,

Some forensic methods have been rooted in science. Medicolegal death investigation emerged from medical science, because death investigation was connected to the protection of public health. Techniques of analytical chemistry were applied to certain types of evidence, such as seized drug analysis, toxicological analysis, and aspects of instrumental analysis applied to trace evidence. More recently, molecular biology gave rise to DNA typing to forensic applications.

The evolution of other forensic disciplines … followed a different course. … Disciplines, such as fingerprints, firearms, and tool marks, blood-stain pattern analysis, tread-impression analysis, and bite mark analysis matured largely outside of the traditional scientific community during a time when admissibility standards for scientific evidence had yet to be formulated. Thus, admissibility of such evidence rightly or wrongly created judicial precedent in decisions that often did not—or could not—involve the level of research that would today be needed to establish scientific validity. …

Professor Bell and her co-authors end the piece with several recommendations, including this one:

Research and academic scientists should become educated about forensic science and take active steps to welcome the discipline into the larger scientific community. A broad effort can help illuminate the causes of failures, help predict when failure is likely to occur, and aid in the development of strategies to mitigate or circumvent those conditions.

The third recent publication places forensic science’s problems in a larger context. On April 17th, the National Association of Scholars released a 70-page report by David Randall and Christopher Welser titled, “The Irreproducibility Crisis of Modern Science: Causes, Consequences, and the Road to Reform.” In it, Randall and Welser explain how:

Improper research techniques, lack of accountability, disciplinary and political groupthink, and a scientific culture biased toward producing positive results together have produced a critical state of affairs. Many supposedly scientific results cannot be reproduced reliably in subsequent investigations, and offer no trustworthy insight into the way the world works.

While the report deals primarily with the natural and social sciences, the authors suggest that,

Federal and state judiciaries should review their treatment of scientific and social-scientific evidence in light of the crisis of reproducibility. …

Judges should make future decisions with a heightened awareness that the crisis of reproducibility has produced a generation or more of presumptively unreliable research. More generally, the judiciary should adopt a standard set of principles for incorporating science into judicial decision-making, perhaps as binding precedent, that explicitly account for the crisis of reproducibility. They should also adopt a standard approach to overturning precedents based on irreproducible science. Finally, a commission of judges should recommend to law schools a required course on science and statistics as they pertain to the law, so as to educate future generations of lawyers and judges about the strengths and weaknesses of statistically-driven research. The commission should also recommend that each state incorporate a science and statistics course into its continuing legal education requirements for attorneys and judges.

These seem like sensible suggestions, and there are many equally sensible suggestions in Erin Murphy’s chapter in Reforming Criminal Justice and in Susan Bell’s PNAS opinion piece. Would implementing some, or even all, of them resolve the forensic science crisis? That’s far from clear. What is clear, however, is that this is a serious problem, and it’s going to take a lot of work to solve it.