Evaluation of LLMs for Process Model Analysis and Optimization
By: Akhil Kumar, Jianliang Leon Zhao, Om Dobariya
Potential Business Impact:
Helps computers find mistakes in business plans.
In this paper, we report our experience with several LLMs for their ability to understand a process model in an interactive, conversational style, find syntactical and logical errors in it, and reason with it in depth through a natural language (NL) interface. Our findings show that a vanilla, untrained LLM like ChatGPT (model o3) in a zero-shot setting is effective in understanding BPMN process models from images and answering queries about them intelligently at syntactic, logic, and semantic levels of depth. Further, different LLMs vary in performance in terms of their accuracy and effectiveness. Nevertheless, our empirical analysis shows that LLMs can play a valuable role as assistants for business process designers and users. We also study the LLM's "thought process" and ability to perform deeper reasoning in the context of process analysis and optimization. We find that the LLMs seem to exhibit anthropomorphic properties.
Similar Papers
From Theory to Practice: Real-World Use Cases on Trustworthy LLM-Driven Process Modeling, Prediction and Automation
Software Engineering
AI helps businesses manage work better.
Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results
Computation and Language
Helps computers create better step-by-step plans.
Domain Adaptation of LLMs for Process Data
Computation and Language
Helps computers predict what happens next in a process.