Limited-data challenge
Medical, language, manufacturing, and niche-domain data can be expensive or impossible to collect at scale. Small datasets increase variance and make memorization look like learning.
Start with transfer learning
Begin from representations learned on larger related corpora, then fine-tune conservatively. Freeze more of model when labels are extremely scarce; unfreeze gradually when validation evidence supports it.
- Choose source domain close to target
- Use lower learning rates during fine-tuning
- Compare frozen-feature baseline before full training
Expand signal, not leakage
Augmentation helps only when transformation preserves label. Text paraphrases, audio perturbations, and image transforms need domain review. Split data before augmentation so variants cannot leak across train and test sets.
- Regularize with dropout and weight decay
- Use stratified cross-validation
- Report uncertainty across seeds
- Inspect errors by subgroup
Practical order
Strong pretrained baseline plus disciplined evaluation usually beats elaborate architecture on limited data.
- Transfer learning
- Domain-valid augmentation
- Regularization
- More representative labels
- Few-shot methods only after baseline