LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks
By: William Fleshman, Benjamin Van Durme
Potential Business Impact:
Lets computers pick the best smart helper for each job.
The proliferation of fine-tuned language model experts for specific tasks and domains signals the need for efficient selection and combination methods. We propose LoRA-Augmented Generation (LAG) for leveraging large libraries of knowledge and task-specific LoRA adapters. LAG requires no additional training or access to data, and efficiently filters, retrieves, and applies experts on a per-token and layer basis. We evaluate LAG on various knowledge-intensive tasks, achieving superior performance over existing data-free methods. We explore scenarios where additional data is available, demonstrating LAG's compatibility with alternative solutions such as retrieval-augmented generation (RAG).
Similar Papers
LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Computation and Language
Makes small AI learn new skills fast.
LAG: Logic-Augmented Generation from a Cartesian Perspective
Computation and Language
Helps computers answer tricky questions correctly.
Semantic-guided LoRA Parameters Generation
Machine Learning (CS)
AI learns new tasks for each person privately.