All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond
By: Shubham Kulkarni , Alexander Lyzhov , Shiva Chaitanya and more
Conversational AI is starting to support real clinical work, but most evaluation methods miss how compliance depends on the full course of a conversation. We introduce Obligatory-Information Phase Structured Compliance Evaluation (OIP-SCE), an evaluation method that checks whether every required clinical obligation is met, in the right order, with clear evidence for clinicians to review. This makes complex rules practical and auditable, helping close the gap between technical progress and what healthcare actually needs. We demonstrate the method in two case studies (respiratory history, benefits verification) and show how phase-level evidence turns policy into shared, actionable steps. By giving clinicians control over what to check and engineers a clear specification to implement, OIP-SCE provides a single, auditable evaluation surface that aligns AI capability with clinical workflow and supports routine, safe use.
Similar Papers
Human-in-the-Loop Interactive Report Generation for Chronic Disease Adherence
Human-Computer Interaction
Doctors get AI help to write patient notes faster.
A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems
Machine Learning (CS)
Makes doctors trust computer health advice.
AI Standardized Patient Improves Human Conversations in Advanced Cancer Care
Human-Computer Interaction
Teaches doctors to talk about hard things better.