# Understanding alpha parameter tuning in LORA paper

I was reading the LORA paper https://arxiv.org/pdf/2106.09685.pdf a thing I don’t understand is section 4.1, where the updates are updated by alpha, where alpha is a constant in r. It is said that alpha is set to the first r tried. Then if I understand correctly, the authors say that this makes it unnecesary to tune the learning rate alpha. I would really appreciate if someone could explain this concept to me. To start with I don’t understand, why the need to scale the weight update by a constant. I mean, all the weights of the updates are optimized in the fine tuning process.

I also wanted to understand why is A initialized randomly and B to zero. Would it make a difference if it would be the other way around (A zero, B random?). Also, what would go wrong if both would be set to zero?

I had the same question. I still haven't got a convincing answer, but while searching I found these that might be helpful:

In this blog, they say:

Alpha scales the learned weights. Existing literature, including the original LoRA paper, generally advises fixing Alpha—often at 16—rather than treating it as a tunable hyperparameter

In literature, they say:

[...] and LoRA alpha is the scaling factor for the weight matrices. The weight matrix is scaled by $$\frac{lora\_alpha}{lora\_rank}$$, and a higher alpha value assigns more weight to the LoRA activations. We chose 16 since this was common practice in training scripts we reviewed and chose a 1:1 ratio so as not to overpower the base model.

While these two passages were clear, I still don't understand why one should scale the update weights. Also, I wouldn't have expected a ratio $$\frac{lora\_alpha}{lora\_rank} > 1$$, but in the tutorial I am following for applying LoRA to Whisper (an ASR model), the ratio is equal to $$2$$, with $$lora\_alpha=64$$ and $$lora\_rank=32$$.

I also wanted to understand why is A initialized randomly and B to zero. Would it make a difference if it would be the other way around (A zero, B random?).

I think they just want initial step of fine tuning to be like we fine tune with only pretrained weight (no additional weight affect the result). One of A or B have to be all zero, so (A zero, B random?) should work too.

Also, what would go wrong if both would be set to zero?

Initializing both of them to zero might cause some issue, I think it is the same reason why we don't initialize deep learning model with zero. The gradient signal sent from A to B will be the same (all zero) and each node in A will look like the same to B.

Related link Initialize perceptron weights with zero

I think lora_alpha/lora_rank is a way of tuning how influential you want finetuning to be. Below is an excerpt from paper which will help explain this better

h = W0x + ∆W x = W0x + BAx

We use a random Gaussian initialization for A and zero for B, so ∆W = BA is zero at the beginning of training. We then scale ∆W x by α/r, where α is a constant in r.

Pay attention the the equation and the last line.

So now if for a certain lora_rank(r) we choose a high lora_alpha(α), the new addition from finetuning is gonna be more influential in comparison to the previous weights as our hidden representation is effectively now h = W0x + ∆W x = W0x + BAxα/r. But if we were to choose a comparatively lower alpha we would be more aligned towards the original pre-trained weights.