How Does LLM Reasoning Work for Code? A Survey and a Call to Action
By: Ira Ceka , Saurabh Pujar , Irene Manotas and more
Potential Business Impact:
Helps computers fix and write computer code.
The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks. These advancements have extended into the domain of code, facilitating complex tasks such as code generation, translation, summarization, and repair. However, their utility for real-world deployment in-the-wild has only recently been studied, particularly on software engineering (SWE) tasks such as GitHub issue resolution. In this study, we examine the code reasoning techniques that underlie the ability to perform such tasks, and examine the paradigms used to drive their performance. Our contributions in this paper are: (1) the first dedicated survey on code reasoning for code tasks, highlighting overarching strategies, hybrid and agentic approaches; (2) a taxonomy of various techniques used to drive code reasoning; (3) a comprehensive overview of performance on common benchmarks and a showcase of new, under-explored benchmarks with high potential in SWE; (4) an exploration on how core properties of code can be used to explain different reasoning techniques; and (5) gaps and potentially under-explored areas for future research.
Similar Papers
Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs
Computation and Language
Makes computers smarter at writing and fixing code.
Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
Software Engineering
Helps computers understand and write computer code better.
Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoning
Machine Learning (CS)
Makes AI smarter by teaching it organized code.