Context Tuning for In-Context Optimization
By: Jack Lu , Ryan Teehan , Zhenbang Yang and more
Potential Business Impact:
Teaches computers to learn from examples faster.
We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of language models (LLMs) without fine-tuning model parameters. While prompt-based adaptation techniques have demonstrated the effectiveness of lightweight adaptation methods for large language models (LLMs), they typically initialize a trainable prompt or prefix with irrelevant tokens for the task at hand. In contrast, Context Tuning initializes the trainable prompt or prefix with task-specific demonstration examples, leveraging the model's inherent In-Context Learning (ICL) ability to extract relevant information for improved few-shot learning performance. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms traditional prompt-based adaptation methods and achieves competitive accuracy to Test-Time Training with significantly higher training efficiency.
Similar Papers
You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Model
Computation and Language
Teaches computers to do many jobs well at once.
Teaching LLMs How to Learn with Contextual Fine-Tuning
Machine Learning (CS)
Teaches computers to learn new things faster.
Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training
Computation and Language
Helps computers understand many languages better.