New Fairness Auditing Method Uses Only Aggregate Data

On Aug 5 2026, Sivan Sabato and Elad Yom‑Tov published a new approach to audit machine-learning classifiers for fairness while respecting privacy constraints ¹. Their work tackles a common obstacle: auditors often cannot inspect a model’s internal logic or access its raw predictions because the classifier may be proprietary or protected by regulations such as HIPAA. The authors therefore ask whether one can still gauge a system’s bias using only aggregated statistics about its outputs across different demographic groups.

The method hinges on the equalized odds criterion—a standard fairness definition that requires a model to have identical true-positive and false-negative rates for all protected subgroups. Sabato and Yom‑Tov translate this into an “unfairness” measure that counts how much of each group experiences deviation from perfect equalized odds. They then derive a computationally efficient procedure that, given only population-level counts of positive predictions and true positives per subgroup, computes the tightest possible lower bound on the overall discrepancy between accuracy and fairness. In other words, even if we never see any single prediction, the algorithm tells us the minimum amount of unfairness that must exist in the system.

Empirical tests confirm the bounds are close to the actual discrepancy in realistic settings. The authors evaluated their technique on several benchmark datasets—including credit-risk and recidivism risk scores—showing that the computed lower bounds were within 5 % of the ground truth when the underlying models were known ¹. This suggests auditors can rely on these aggregate-only estimates to flag potentially biased systems before they reach deployment.

Beyond the technical contribution, the paper highlights practical implications. By not requiring individual data, it sidesteps privacy concerns that often block audits of sensitive applications such as healthcare or criminal justice. It also offers a way to audit legacy systems where source code is unavailable but aggregated performance metrics (e.g., confusion matrix entries) are publicly released. However, the method’s assumptions deserve attention: it presumes independence among features inside protected subgroups—a condition that may fail for complex real-world data sets. Auditors should therefore test this assumption empirically before trusting the bounds alone.

The approach echoes earlier calls from the AI ethics community to develop tools that can certify fairness without compromising user confidentiality. As noted by experts, the equalized odds criterion directly addresses disparities highlighted in ProPublica’s investigation into COMPAS recidivism risk scores in 2016, which revealed disparate impact across racial groups.

In summary, Sabato and Yom‑Tov provide a concrete step toward fairer AI systems. Their aggregate-statistics auditing framework could become part of standard compliance checklists, especially for regulated sectors where model internals cannot be inspected. Future work will likely explore relaxing the independence assumption and extending the technique to multi-class classifiers. The development underscores an emerging trend: AI researchers increasingly design methods that respect both ethical constraints and practical deployment realities.