LoRMA: Low-Rank Multiplicative Adaptation for LLMs
By: Harsh Bihany, Shubham Patel, Ashutosh Modi
Potential Business Impact:
Makes smart computer programs learn faster and cheaper.
Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.
Similar Papers
Less is More: Resource-Efficient Low-Rank Adaptation
Computation and Language
Makes AI learn faster and better with less effort.
ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
Machine Learning (CS)
Makes smart computer programs learn faster and better.
TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models
Machine Learning (CS)
Makes AI learn new things with less effort.