From Text to Trajectories: GPT-2 as an ODE Solver via In-Context
By: Ziyang Ma , Baojian Zhou , Deqing Yang and more
Potential Business Impact:
Teaches computers to solve math problems with examples.
In-Context Learning (ICL) has emerged as a new paradigm in large language models (LLMs), enabling them to perform novel tasks by conditioning on a few examples embedded in the prompt. Yet, the highly nonlinear behavior of ICL for NLP tasks remains poorly understood. To shed light on its underlying mechanisms, this paper investigates whether LLMs can solve ordinary differential equations (ODEs) under the ICL setting. We formulate standard ODE problems and their solutions as sequential prompts and evaluate GPT-2 models on these tasks. Experiments on two types of ODEs show that GPT-2 can effectively learn a meta-ODE algorithm, with convergence behavior comparable to, or better than, the Euler method, and achieve exponential accuracy gains with increasing numbers of demonstrations. Moreover, the model generalizes to out-of-distribution (OOD) problems, demonstrating robust extrapolation capabilities. These empirical findings provide new insights into the mechanisms of ICL in NLP and its potential for solving nonlinear numerical problems.
Similar Papers
Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics
Machine Learning (CS)
Computers predict future events from math patterns.
Towards Foundation Inference Models that Learn ODEs In-Context
Machine Learning (CS)
Helps computers learn how things change from messy data.
Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study
Machine Learning (CS)
Teaches computers to learn better from examples.