What NIST Data Shows About Facial Recognition and Demographics

In December 2019, the National Institute of Standards and Technology (NIST) published the most comprehensive report to date on the performance of facial recognition algorithms – the core component of facial recognition technology – across race, gender and other demographic groups. The most significant takeaway from the NIST report is that it confirms current facial recognition technology performs far more effectively across racial and other demographic groups than had been widely reported. 

Headlines over the past year have highlighted sincere concerns that use of facial recognition technology might negatively impact women and minorities. Industry is striving to provide technology that is as effective and accurate as possible across all types of uses, deployment settings and demographic characteristics in order to fully address these concerns.Both developers and end users have a responsibility to minimize any negative effects that could result when the technology does not perform as intended though proper design, configuration, policies and procedures. SIA strongly believes that facial recognition makes our country safer and brings value to our everyday lives when used effectively and responsibly. No technology product should ever be used for purposes that are unlawful, unethical or discriminatory.

NIST’s December report is the most complete scientific evaluation yet of the performance of current facial recognition technology across demographic variables, testing 189 algorithms provided by 99 developers using 18 million images of 8 million people contained in four different types of data sets.

NIST found that, “to the extent there are demographic differentials,” error rates in verification-type algorithms are “absolutely low,” generally below 1 percent and many below 0.5 percent. Even more significantly, NIST found that in the most accurate identification algorithms it tested, differences in performance across demographic groups were “undetectable.” Notably for policymakers, the most well-known U.S. government applications already use some of the highest-performing technologies. The report specifically identifies six suppliers of identification algorithms with undetectable differences in “false positive” rates. 

Overall, modern facial recognition technology is highly accurate. NIST has documented massive improvements in recent years, noting in 2018 the software tested was at least 20 times more accurate than it was in 2014, and in 2019 finding “close to perfect” performance by high-performing algorithms with miss rates averaging 0.1 percent. On this measurement, the accuracy of facial recognition is reaching that of automated fingerprint comparison, which is generally viewed as the gold standard for identification.

A human analyst will play a critical role in use of facial recognition as a tool in law enforcement investigations or as part of any process with potential high-consequence outcomes for individuals. NIST has documented that the most accurate identification results occur when facial recognition is combined with trained human review versus either element alone, which may explain U.S. law enforcement’s decade-plus operating history without any example of it contributing to a mistaken arrest or imprisonment. Also, false positives may be naturally limited by the size of the data set used.

Applied to security solutions developed by our industry, biometric technologies like facial recognition increase the effectiveness of safety and security measures that protect people from harm. Any significant bias in technology performance makes it harder to achieve this goal. While the NIST report provides a snapshot of the performance of current technology, it also provides a critical benchmark against which developers will work to improve the technology, as industry progress is tracked through the ongoing Facial Recognition Vendor Test (FRVT) program.

Read more of SIA’s analysis of the NIST demographic effects report here.