Fairness Audit Method Uses Only Population Statistics

A team of researchers published their findings on August 5, 2026 in the Journal of Artificial Intelligence Research (JAIR) ¹. The method estimates how far a black‑box classifier deviates from ideal fair behavior using only aggregated population data.

The authors define an “unfairness” metric based on the equalized odds fairness criterion. It quantifies the fraction of each protected subpopulation on which a classifier’s positive prediction rate differs from the true positive rate. By focusing on aggregate rates rather than individual outcomes, the technique sidesteps privacy concerns while still revealing systemic bias. Using this measure they develop a computationally efficient procedure that calculates the tightest possible lower bound on a classifier’s discrepancy – a value that jointly captures both inaccuracy and unfairness. The calculation requires only two statistics per group: the overall proportion of positive predictions and the proportion of actual positives.

Empirical evaluations confirm the tightness of the proposed lower bound in practical settings. In experiments across several datasets, the algorithm produced bounds close to real discrepancies. The method proves effective for various classifiers, demonstrating broad applicability.

This work directly addresses concerns raised about auditing opaque AI systems following ProPublica’s investigation into the COMPAS recidivism risk assessment tool in 2016 ². That case highlighted how lack of access to proprietary algorithms can hinder independent verification. By relying solely on population-level statistics, Sabato and Yom‑Tov offer an accessible audit path even without model internals or individual data points. Moreover, the authors suggest that their framework could be extended using statistical discrimination detection tools developed by Kleinberg et al., further strengthening its utility in regulatory contexts.

Because it requires only aggregated rates, organizations can apply it without compromising sensitive user information. The approach offers a less invasive strategy suitable for black-box scenarios.

In sum, Sabato and Yom‑Tov’s contribution provides a practical tool for measuring classifier fairness when internal inspection is impossible.