Hello r/learnmachinelearning!

I’ve been thinking about when to use LoRAs versus full fine-tuning, and I wanted to check if my understanding is valid.

My Understanding of LoRAs:

LoRAs seem most useful when there exists a manifold in the model that humans would associate with a concept, but the model hasn’t properly learned the connection.

Example: A model trained on « red » and « truck » separately might struggle with « red truck » (where f(red + truck) ≠ red truck), even though a red truck manifold exists within the model’s latent space. By training a « red truck » LoRA, we’re teaching the model that f(red + truck) should map to that existing red truck manifold.

LoRAs vs. Full Fine-Tuning:

LoRAs: Create connections to existing manifolds in the model Full Fine-Tuning: Can potentially create entirely new manifolds that didn’t previously exist

Practical Implication:

If we could determine whether a manifold for our target concept already exists in the model, we could make an informed decision about whether:

A LoRA would be sufficient (if the manifold exists) Full fine-tuning is necessary (if we need to create a new manifold)

Does this reasoning make sense? Any thoughts or corrections would be appreciated!