The 2009 NAS Report and Forensic Science Reliability

The findings were damning. After reviewing decades of research, practice, and validation data, the committee concluded that many of the forensic disciplines routinely admitted in American courts lacked the scientific foundation necessary to support the conclusions drawn from them. Courts had been admitting testimony that overstated what the underlying science could demonstrate, many disciplines had no meaningful research base establishing their reliability, and the adversarial system had failed to police the problem.

The report is not ancient history. It remains the most comprehensive independent assessment of forensic science reliability ever conducted, its findings were confirmed and extended by later research, and its conclusions bear directly on criminal cases in Texas courts today.

What the Report Found

Lack of scientific validation. The committee found that many pattern-matching disciplines, including bite mark analysis, microscopic hair comparison, tool mark and firearms identification, and footwear analysis, had never been subjected to the rigorous empirical testing that would establish their accuracy, error rates, and reliability. The report stated plainly that the interpretation of forensic evidence is often not grounded in studies validating it. Disciplines were being used to convict defendants on assumptions that had never been scientifically tested.

No known error rates. For most forensic disciplines there were no established error rates, no systematic measurement of how often examiners are wrong, and no requirement that the jury be told. An examiner who testifies that a latent fingerprint “matches” a suspect to the exclusion of everyone else is making a statistical claim that cannot be evaluated without knowing the method’s false-positive rate, and for most disciplines that rate was simply unknown because no one had measured it. We take up the fingerprint version of this problem in our post on whether a fingerprint match is really a match.

Subjective judgments presented as objective certainty. Many forensic methods rest on subjective judgments by trained examiners. The report found that this subjectivity was rarely acknowledged in court, where examiners presented conclusions with a degree of certainty the methodology did not support. An examiner who says a bite mark was “made by” a specific person to “reasonable forensic certainty” asserts a precision the discipline has never shown it can achieve.

Fragmentation and weak oversight. Forensic science in the United States was fragmented across thousands of laboratories operating under widely varying standards, with no uniform accreditation, no standardized training, and few mechanisms for independent quality assurance. Laboratories housed inside law enforcement agencies operated without the independence from prosecutorial pressure that scientific reliability requires. What an outside audit can expose is the subject of our post on forensic laboratory audits, and what happens when a single trusted analyst goes unchecked is the subject of our post on the Missy Woods DNA scandal.

The courts did not catch it. The report found that the legal system’s own safeguards, the rules meant to screen scientific evidence together with cross-examination and adversarial testing, had largely failed to identify and exclude unreliable forensic science. Once a method was admitted in one jurisdiction, courts elsewhere routinely admitted it without independent scrutiny. Precedent substituted for scientific validation.

The Specific Disciplines Criticized

The report singled out several disciplines with significant reliability problems:

• Microscopic hair analysis. Matching crime-scene hairs to a suspect by visual comparison under a microscope had never been validated as a reliable identification method. A later FBI review of cases using hair comparison confirmed error rates that had contributed to wrongful convictions, and hair analysis is now generally considered unsuitable for identification testimony.
• Bite mark analysis. The foundational premises, that human dentition is unique and that skin reliably records dental impressions, had not been scientifically established. Bite mark evidence has since been tied to a string of wrongful convictions, a story we tell through one man’s 27 years on death row in our post on bite-mark evidence.
• Firearms and tool mark identification. The research base was insufficient to support the categorical claim examiners routinely make, that a bullet or cartridge case was fired by one specific weapon to the exclusion of all others.
• Blood spatter analysis. Interpretation lacked the scientific study needed to support the precision with which conclusions were often presented, a question we take up in our post on how reliable blood spatter analysis is.

What the NAS Report Did Not Criticize

Just as important, the report distinguished these unreliable disciplines from well-validated ones. It did not attack forensic science wholesale. DNA analysis of single-source samples was found to rest on a strong scientific foundation, with an empirically validated methodology, measurable error rates, and established statistical interpretation. Toxicology and blood-alcohol analysis performed on validated instruments were likewise found to be scientifically sound, with the caveat that even a reliable method can produce an unreliable result when it is not properly executed. That caveat is where much of the modern fight lives: even validated DNA science now runs on complex probabilistic software whose outputs can diverge, which we examine in our post on probabilistic genotyping and TrueAllele.

How Later Research Confirmed It

The NAS report was not the last word; it was closer to the first. In 2016, the President’s Council of Advisors on Science and Technology, known as PCAST, issued its own report on feature-comparison methods and reached compatible conclusions. Reviewing the empirical literature, PCAST found that several pattern-matching disciplines still lacked what it called foundational validity, that too few properly designed studies existed to measure their error rates, and that where error rates had been measured, they were far higher than courtroom testimony implied. PCAST concluded that bite mark comparison was not scientifically reliable, and that even latent fingerprint analysis, though better supported, carried a false-positive rate that had to be disclosed rather than hidden behind claims of certainty. More than fifteen years on, the core lesson of the 2009 report has been reinforced, not retired.

Why It Still Matters in Texas Courts

The NAS report does not bind Texas courts as a matter of law, but it remains the authoritative scientific assessment of forensic reliability, and Texas has its own tool for putting that science to the test. Texas criminal courts do not apply the federal Daubert standard; they apply the reliability standard from Kelly v. State, which requires the party offering scientific evidence to establish three things: that the underlying theory is valid, that the technique applying the theory is valid, and that the technique was properly applied on the occasion in question. The proponent carries that burden by clear and convincing evidence, and a defendant can request a reliability hearing to force the issue.

In any case where the State relies on pattern-matching evidence, for example bite marks, hair, tool marks, firearms, or footwear, the NAS and PCAST findings go directly to both admissibility and weight. A defense attorney who can put those findings in front of a judge at a reliability hearing, or in front of a jury on cross-examination, is engaging the scientific validity of the State’s evidence at the level it deserves.

Texas has also written responses to exactly these problems into its law. It created the Texas Forensic Science Commission in 2005 to investigate professional negligence and misconduct across forensic disciplines, it became the first state to recommend a moratorium on bite mark evidence, and its junk-science writ, Article 11.073 of the Code of Criminal Procedure, lets a convicted person seek relief when the science underlying a conviction has been discredited or has changed. The 2009 report is the intellectual foundation for much of that machinery.

If forensic science evidence is central to your case, contact Deandra Grant Law for a free, confidential case review. Managing Partner Deandra Grant and Partner Douglas Huff both hold the ACS-CHAL Forensic Lawyer-Scientist designation, and Deandra Grant also holds an M.S. in Pharmaceutical Science and a Graduate Certificate in Forensic Toxicology. Call (214) 225-7117 or visit texasdwisite.com.

This post is general legal information for educational purposes, not legal advice. Citations should be verified against current law before being relied on in any filing.

Sources and Further Reading

• National Research Council, “Strengthening Forensic Science in the United States: A Path Forward” (2009), the full NAS report.
• President’s Council of Advisors on Science and Technology, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016).
• Kelly v. State, 824 S.W.2d 568 (Tex. Crim. App. 1992), the Texas reliability standard; and Tex. Code Crim. Proc. Art. 11.073, the junk-science writ.
• Related posts in this series: fingerprint reliability, bite-mark evidence, TrueAllele and the right to inspect the code, probabilistic DNA software, fingerprints recovered from fired casings, whether DNA results can be hacked, DNA methylation clocks, forensic laboratory audits, blood spatter analysis, and the Missy Woods DNA scandal.