04 · CAPACITY
VerifiedRank, Alpha, and Target Modules
Separate capacity, scale, and intervention location; change one variable at a time.
Rank r determines adapter capacity and the r × (d_in + d_out) parameter budget. Alpha controls alpha/r in standard LoRA and alpha/√r in rsLoRA.
Target modules decide which real attention and MLP matrices receive adapters. Names are architecture-specific and must be verified on the model.
01
First thought
The same alpha/r means the same adapter capacity.
02
Correction
Effective scale may match while a higher rank still carries more parameters and capacity.
03
Decision rule
Start at r=8/alpha=8; if capacity is insufficient, hold everything else fixed and test r=16/alpha=16.