06 · GENERALIZATION
VerifiedLoss, Overfitting, and Forgetting
Do not confuse loss with business quality; combine domain, format, safety, and retention metrics.
Training loss tracks fit to the optimized target; validation loss tracks generalization on an unseen split. Falling train loss with rising validation loss is an overfitting signal.
Catastrophic forgetting is different: the new domain improves while prior/general ability declines. LoRA preserves the base but an active adapter can still create behavioral interference.
First thought
Lower validation loss always means the better production model.
Correction
Loss measures the token objective; JSON validity, task accuracy, and safety require separate metrics.
Decision rule
Select checkpoints with validation loss + task metric + retention threshold.