Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines
By: Jean Seo , Gibaeg Kim , Kihun Shin and more
Potential Business Impact:
Helps AI doctors understand patient problems better.
We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings.
Similar Papers
Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines
Computation and Language
Helps AI doctors understand patient problems better.
From EMR Data to Clinical Insight: An LLM-Driven Framework for Automated Pre-Consultation Questionnaire Generation
Artificial Intelligence
Creates personalized health forms from patient records.
Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
Computation and Language
Helps doctors write patient notes faster.