A New Perspective on Precision and Recall for Generative Models
By: Benjamin Sykes , Loïc Simon , Julien Rabin and more
Potential Business Impact:
Helps check if computer-made pictures are good.
With the recent success of generative models in image and text, the question of their evaluation has recently gained a lot of attention. While most methods from the state of the art rely on scalar metrics, the introduction of Precision and Recall (PR) for generative model has opened up a new avenue of research. The associated PR curve allows for a richer analysis, but their estimation poses several challenges. In this paper, we present a new framework for estimating entire PR curves based on a binary classification standpoint. We conduct a thorough statistical analysis of the proposed estimates. As a byproduct, we obtain a minimax upper bound on the PR estimation risk. We also show that our framework extends several landmark PR metrics of the literature which by design are restrained to the extreme values of the curve. Finally, we study the different behaviors of the curves obtained experimentally in various settings.
Similar Papers
On the Geometry of Receiver Operating Characteristic and Precision-Recall Curves
Machine Learning (CS)
Helps computers choose the best way to guess.
Probabilistic Robustness for Free? Revisiting Training via a Benchmark
CV and Pattern Recognition
Tests computer smarts against tricky mistakes.
Non-Parametric Probabilistic Robustness: A Conservative Metric with Optimized Perturbation Distributions
CV and Pattern Recognition
Makes AI more trustworthy with unknown errors.