Dissecting Atomic Facts: Visual Analytics for Improving Fact Annotations in Language Model Evaluation
By: Manuel Schmidt, Daniel A. Keim, Frederik L. Dennig
Potential Business Impact:
Helps check if AI is telling the truth.
Factuality evaluation of large language model (LLM) outputs requires decomposing text into discrete "atomic" facts. However, existing definitions of atomicity are underspecified, with empirical results showing high disagreement among annotators, both human and model-based, due to unresolved ambiguity in fact decomposition. We present a visual analytics concept to expose and analyze annotation inconsistencies in fact extraction. By visualizing semantic alignment, granularity and referential dependencies, our approach aims to enable systematic inspection of extracted facts and facilitate convergence through guided revision loops, establishing a more stable foundation for factuality evaluation benchmarks and improving LLM evaluation.
Similar Papers
Improving Reliability and Explainability of Medical Question Answering through Atomic Fact Checking in Retrieval-Augmented LLMs
Computation and Language
Makes AI doctors give correct, trustworthy medical answers.
AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
Computation and Language
Checks if AI's answers are true or made up.
Fact in Fragments: Deconstructing Complex Claims via LLM-based Atomic Fact Extraction and Verification
Artificial Intelligence
Checks if stories are true, even tricky ones.