From Technical Excellence to Practical Adoption: Lessons Learned Building an ML-Enhanced Trace Analysis Tool
By: Kaveh Shahedi , Matthew Khouzam , Heng Li and more
Potential Business Impact:
Makes debugging tools trusted and easy to adopt
System tracing has become essential for understanding complex software behavior in modern systems, yet sophisticated trace analysis tools face significant adoption gaps in industrial settings. Through a year-long collaboration with Ericsson Montr\'eal, developing TMLL (Trace-Server Machine Learning Library, now in the Eclipse Foundation), we investigated barriers to trace analysis adoption. Contrary to assumptions about complexity or automation needs, practitioners struggled with translating expert knowledge into actionable insights, integrating analysis into their workflows, and trusting automated results they could not validate. We identified what we called the Excellence Paradox: technical excellence can actively impede adoption when conflicting with usability, transparency, and practitioner trust. TMLL addresses this through adoption-focused design that embeds expert knowledge in interfaces, provides transparent explanations, and enables incremental adoption. Validation through Ericsson's experts' feedback, Eclipse Foundation's integration, and a survey of 40 industry and academic professionals revealed consistent patterns: survey results showed that 77.5% prioritize quality and trust in results over technical sophistication, while 67.5% prefer semi-automated analysis with user control, findings supported by qualitative feedback from industrial collaboration and external peer review. Results validate three core principles: cognitive compatibility, embedded expertise, and transparency-based trust. This challenges conventional capability-focused tool development, demonstrating that sustainable adoption requires reorientation toward adoption-focused design with actionable implications for automated software engineering tools.
Similar Papers
Engineering Trustworthy Automation: Design Principles and Evaluation for AutoML Tools for Novices
Human-Computer Interaction
Helps beginners build smart computer programs easily.
Using LLMs and Essence to Support Software Practice Adoption
Software Engineering
Helps software teams follow best practices.
Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning
Software Engineering
Fixes teamwork problems for big tech projects.