Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications in AI-assisted Data Analysis
By: Qianou Ma, Kenneth Koedinger, Tongshuang Wu
Potential Business Impact:
Helps students use AI for coding better.
LLMs promise to democratize technical work in complex domains like programmatic data analysis, but not everyone benefits equally. We study how students with varied expertise use LLMs to complete Python-based data analysis in computational notebooks in a non-major course. Drawing on homework logs, recordings, and surveys from 36 students, we ask: Which expertise matters most, and how does it shape AI use? Our mixed-methods analysis shows that technical expertise -- not AI familiarity or communication skills -- remains a significant predictor of success. Students also vary widely in how they leverage LLMs, struggling at stages of forming intent, expressing inputs, interpreting outputs, and assessing results. We identify success and failure behaviors, such as providing context or decomposing prompts, that distinguish effective use. These findings inform AI literacy interventions, highlighting that lightweight demonstrations improve surface fluency but are insufficient; deeper training and scaffolds are needed to cultivate resilient AI use skills.
Similar Papers
Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses
Computers and Society
Helps students get homework help anytime, anywhere.
Prompting the Professoriate: A Qualitative Study of Instructor Perspectives on LLMs in Data Science Education
Computers and Society
Teachers use AI to help students learn better.
A Survey of LLM-Based Applications in Programming Education: Balancing Automation and Human Oversight
Computers and Society
Helps students learn coding with smart computer tutors.