What Happened
A common recommendation in QLoRA tutorials is to start with a learning rate of 2e-4. However, this guidance may be misleading for those working with smaller datasets, typically around 5,000 to 10,000 samples. Fine-tuning at this rate on limited data often leads to overfitting, where the model performs well on the training data but poorly on evaluation metrics.
Why It Matters
This issue is significant because many practitioners rely on the standard learning rate without considering the size of their datasets. When using 2e-4 for small datasets, users might observe declining training loss while evaluation loss stagnates or worsens. This results in wasted time and effort, causing frustration as individuals may incorrectly blame their data quality or other factors for poor performance.
Context
The 2e-4 learning rate originates from research involving larger datasets, like Alpaca, which contains 52,000 samples. The assumption is that this rate works well at scale, but it has not been well-documented how it translates to smaller datasets. Many tutorials and documentation, including those from Unsloth, suggest 2e-4 as a 'starting point' but fail to highlight the need for adjustment based on dataset size.
What It Means
Practitioners should reconsider the default learning rate of 2e-4 when fine-tuning models on smaller datasets. A learning rate of 1e-4 or lower, combined with increased epochs, may yield better results. This simple adjustment can lead to significant improvements in evaluation metrics, as evidenced by anecdotal experiences from users who have successfully adjusted their rates. If there's ongoing research supporting the flat rate of 2e-4 for small datasets, it remains largely undiscovered, leaving many to struggle unnecessarily with their models.
