arXiv 2410.21228
LoRA vs Full Fine-tuning: An Illusion of Equivalence
By Reece Shuttleworth, Jacob Andreas, et al.
Published 2024-10-28
Discussion
Read the public discussion and references gathered around this paper.
Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, are their learned solutions really equivalent? We study how LoRA and full-finetuning change pre-trained models by analyzing the model's weight matrices through the l…