FBI Publishes Major Handwriting Analysis Study - Use With Caution

Forensic handwriting examination involves the comparison of writing samples by forensic document examiners (FDEs) to determine whether or not they were written by the same person. The FBI’s Laboratory Division, in conjunction with Noblis, Inc., published a major scientific research paper today in the Proceedings of the National Academy of Sciences about the accuracy and reliability of forensic handwriting comparison. The study is the largest of its kind, involving more than 80 document examiners from U.S. and international crime laboratories and private practice. Collectively, these examiners made more than 7,000 document comparisons and provided information with which to correlate results to levels of education and experience, along with other metadata.

The paper, “Accuracy and Reliability of Forensic Handwriting Comparisons,” summarizes the results of a five-year project to examine how often forensic handwriting examiners reach correct conclusions when determining whether a document was written by a specific individual by comparing it to samples of known handwriting from that person. The FBI Laboratory undertook this research to provide estimates of error rates—how often document examiners make correct writership decisions—as well as how often an examiner reaches the same conclusion when seeing the same documents again, and how often other examiners reach the same conclusions.

 In the study, eighty-six practicing FDEs each conducted up to 100 handwriting comparisons, resulting in 7,196 conclusions on 180 distinct comparison sets, using a five-level conclusion scale. Erroneous “written by” conclusions (false positives) were reached in 3.1% of the nonmated comparisons, while 1.1% of the mated comparisons yielded erroneous “not written by” conclusions (false negatives). False positive rates were markedly higher for nonmated samples written by twins (8.7%) compared to nontwins (2.5%). 

Notable associations between training and performance were observed: FDEs with less than 2 y of formal training generally had higher error rates, but they also had higher true positive and true negative rates because they tended to provide more definitive conclusions; FDEs with at least 2 y of formal training were less likely to make definitive conclusions, but those definitive conclusions they made were more likely to be correct (higher positive predictive and negative predictive values). The authors did not observe any association between writing style (cursive vs. printing) and rates of errors or incorrect conclusions. The report also provides details on the repeatability and reproducibility of conclusions, and reports how conclusions are affected by the quantity of writing and the similarity of content.

“This large-scale study demonstrates the commitment of the FBI and our partners to critically evaluate forensic examinations widely used throughout the criminal justice system,” said FBI Director Christopher Wray. “Using scientific rigor to affirm the accuracy and reliability of forensic techniques helps protect the integrity of our judicial process and ensure that everyone is treated fairly under the law.”

However, as we have seen with the Pandemic, the problem is not the science, but those who weaponize it, an inside job, or a process flow:  For 2000-plus years we have been warned about the risk of using lawyers, as they are mostly interested in arriving at the best argument money can buy, not uncovering the truth.  This study just feeds into that narrative:  Figures (Science) Dont Lie - Liars figure.