api: YAML JSON TOON Database
version: 1.0.0
format: json
dataset:
  id: 62
  slug: regularization-techniques
  title: Regularization Techniques in Machine Learning
  description: "Techniques to prevent overfitting: L1/L2 regularization, dropout, batch normalization, early stopping, data augmentation, label smoothing, and mixup."
  summary: "Regularization techniques that reduce overfitting, with formula, effect, uses and notes: L1 and L2 penalties, dropout, batch normalization, early stopping, data augmentation, label smoothing and mixup or CutMix."
  use_when[2]: Your model fits training data well but generalizes poorly.,You are choosing regularization for a neural network or linear model.
  caveats[2]: Regularization has hyperparameters that need tuning; too much causes underfitting.,"Batch normalization's regularizing effect is a side effect, not its main purpose."
  related[4]: deep-learning-optimizers,ml-training-algorithms,ml-evaluation-metrics,ai-ethics-interpretability
  category: AI & ML Terms
  category_slug: ai-ml-terms
  tags: "regularization,overfitting,dropout,batch-norm,l1,l2,early-stopping,data-augmentation,label-smoothing"
  view_count: 0
  created_at: 1778695229
  updated_at: 1778695229
  entry_count: 8
  fields[6]{section,field,type,example}:
    techniques,name,string,L1 Regularization (Lasso)
    techniques,formula,string,Loss + λ * Σ|w_i|
    techniques,effect,string,Drives weights to exactly zero (sparse solution…
    techniques,use_cases,array,"[4 items]"
    techniques,notes,string,L1 penalty creates sparse weight vectors; usefu…
    techniques,mechanism,string,Randomly set fraction p of activations to zero …
data:
  techniques[8]:
    - name: L1 Regularization (Lasso)
      formula: Loss + λ * Σ|w_i|
      effect: Drives weights to exactly zero (sparse solutions)
      use_cases[4]: Feature selection,Sparse models,Compressed sensing,When many irrelevant features
      notes: L1 penalty creates sparse weight vectors; useful for automatic feature selection
    - name: L2 Regularization (Ridge)
      formula: Loss + λ * Σw_i²
      effect: Shrinks weights toward zero but not exactly zero (smooth solutions)
      use_cases[3]: General-purpose regularization,Multicollinearity mitigation,Default regularization for most models
      notes: Most common form; equivalent to Gaussian prior on weights; differentiable everywhere
    - name: Dropout
      mechanism: Randomly set fraction p of activations to zero during training
      effect: Prevents co-adaptation of neurons; ensemble of thinned networks
      use_cases[3]: Fully connected layers in deep networks,When overfitting is observed,Standard in most deep learning architectures
      notes: "Typical rates: 0.2-0.5 for FC layers, 0.1-0.2 for conv layers; scale activations by 1/(1-p) at test time (or use inverted dropout)"
    - name: Batch Normalization
      mechanism: "Normalize activations per mini-batch: x̂ = (x - μ_batch) / √(σ²_batch + ε); y = γx̂ + β"
      effect: Reduces internal covariate shift; allows higher learning rates; slight regularization
      use_cases[4]: Deep CNNs,Very deep networks,When training is unstable,Enabling higher learning rates
      notes: Maintains running mean/variance for inference; can hurt performance with small batches or RNNs
    - name: Early Stopping
      mechanism: Monitor validation loss; stop training when val_loss stops improving for patience epochs
      effect: Prevents overfitting by stopping before model memorizes training data
      use_cases[3]: Any iterative training algorithm,When validation set available,"Neural networks, gradient boosting"
      notes: Save best model weights during training; restore after stopping; patience=5-20 typical
    - name: Data Augmentation
      mechanism: "Apply label-preserving transformations to training data (rotation, flip, crop, color jitter for images)"
      effect: Increases effective training set size; teaches invariances
      use_cases[4]: Computer vision (standard practice),"NLP (synonym replacement, back-translation)","Audio (pitch shift, time stretch)",When data is limited
      notes: Most effective regularization for vision; AutoAugment and RandAugment automate policy search
    - name: Label Smoothing
      mechanism: "Replace hard labels (0/1) with soft targets: y_smooth = (1 - ε) * y_hard + ε / K"
      effect: Prevents overconfidence; improves calibration; reduces gap between train and test accuracy
      use_cases[4]: "Image classification (ResNet, EfficientNet)",Machine translation,When model is overconfident,Knowledge distillation teacher
      notes: Typical ε = 0.1; prevents model from assigning probability 1 to any class; improves generalization
    - name: Mixup / CutMix
      mechanism: "Mixup: interpolate pairs of images and labels. CutMix: cut and paste patches"
      effect: Linearizes training between examples; reduces memorization
      use_cases[3]: Image classification,When stronger regularization needed,State-of-the-art vision models
      notes: "Mixup: (x', y') = (λx_i + (1-λ)x_j, λy_i + (1-λ)y_j); CutMix often performs better by preserving local structure"