Continual-learning for Modelling Low-Resource Languages from Large Language Models
By: Santosh Srinath K , Mudit Somani , Varun Reddy Padala and more
Potential Business Impact:
Keeps AI smart when learning new languages.
Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.
Similar Papers
Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
Computation and Language
Keeps AI from forgetting speech skills during training.
When Continue Learning Meets Multimodal Large Language Model: A Survey
Machine Learning (CS)
Helps AI learn new things without forgetting old ones.
Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
CV and Pattern Recognition
Helps AI learn new things without forgetting old ones.