Decomposing Global AUC into Cluster-Level Contributions for Localized Model Diagnostics
By: Agus Sudjianto, Alice J. Liu
Potential Business Impact:
Finds hidden problems in computer predictions.
The Area Under the ROC Curve (AUC) is a widely used performance metric for binary classifiers. However, as a global ranking statistic, the AUC aggregates model behavior over the entire dataset, masking localized weaknesses in specific subpopulations. In high-stakes applications such as credit approval and fraud detection, these weaknesses can lead to financial risk or operational failures. In this paper, we introduce a formal decomposition of global AUC into intra- and inter-cluster components. This allows practitioners to evaluate classifier performance within and across clusters of data, enabling granular diagnostics and subgroup analysis. We also compare the AUC with additive performance metrics such as the Brier score and log loss, which support decomposability and direct attribution. Our framework enhances model development and validation practice by providing additional insights to detect model weakness for model risk management.
Similar Papers
An Estimand-Focused Approach for AUC Estimation, Generalization, and Comparison: From Non-representative Samples to Target Population
Methodology
Makes health tests work fairly everywhere.
Behavior of prediction performance metrics with rare events
Machine Learning (Stat)
Helps doctors predict rare sicknesses more accurately.
SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers
Machine Learning (CS)
Finds when computer programs fail for certain people.