Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
By: Chenxu Wang , Yilin Lyu , Zicheng Sun and more
Potential Business Impact:
Teaches AI new things without forgetting old ones.
Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.
Similar Papers
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Models
Machine Learning (CS)
Helps AI learn new things without forgetting old ones.
GoRA: Gradient-driven Adaptive Low Rank Adaptation
Machine Learning (CS)
Makes AI learn faster and better.
Merge before Forget: A Single LoRA Continual Learning via Continual Merging
Machine Learning (CS)
Merges AI learning without forgetting old lessons.