api: YAML JSON TOON Database
version: 1.0.0
format: json
dataset:
  id: 62
  slug: regularization-techniques
  title: Regularization Techniques in Machine Learning
  description: "Techniques to prevent overfitting: L1/L2 regularization, dropout, batch normalization, early stopping, data augmentation, label smoothing, and mixup."
  summary: "Regularization techniques that reduce overfitting, with formula, effect, uses and notes: L1 and L2 penalties, dropout, batch normalization, early stopping, data augmentation, label smoothing and mixup or CutMix."
  use_when:
    - Your model fits training data well but generalizes poorly.
    - You are choosing regularization for a neural network or linear model.
  caveats:
    - Regularization has hyperparameters that need tuning; too much causes underfitting.
    - Batch normalization's regularizing effect is a side effect, not its main purpose.
  related:
    - deep-learning-optimizers
    - ml-training-algorithms
    - ml-evaluation-metrics
    - ai-ethics-interpretability
  category: AI & ML Terms
  category_slug: ai-ml-terms
  tags: regularization,overfitting,dropout,batch-norm,l1,l2,early-stopping,data-augmentation,label-smoothing
  view_count: 0
  created_at: 1778695229
  updated_at: 1778695229
  entry_count: 8
  fields:
    - section: techniques
      field: name
      type: string
      example: L1 Regularization (Lasso)
    - section: techniques
      field: formula
      type: string
      example: Loss + λ * Σ|w_i|
    - section: techniques
      field: effect
      type: string
      example: Drives weights to exactly zero (sparse solution…
    - section: techniques
      field: use_cases
      type: array
      example: "[4 items]"
    - section: techniques
      field: notes
      type: string
      example: L1 penalty creates sparse weight vectors; usefu…
    - section: techniques
      field: mechanism
      type: string
      example: Randomly set fraction p of activations to zero …
data:
  techniques:
    - name: L1 Regularization (Lasso)
      formula: Loss + λ * Σ|w_i|
      effect: Drives weights to exactly zero (sparse solutions)
      use_cases:
        - Feature selection
        - Sparse models
        - Compressed sensing
        - When many irrelevant features
      notes: L1 penalty creates sparse weight vectors; useful for automatic feature selection
    - name: L2 Regularization (Ridge)
      formula: Loss + λ * Σw_i²
      effect: Shrinks weights toward zero but not exactly zero (smooth solutions)
      use_cases:
        - General-purpose regularization
        - Multicollinearity mitigation
        - Default regularization for most models
      notes: Most common form; equivalent to Gaussian prior on weights; differentiable everywhere
    - name: Dropout
      mechanism: Randomly set fraction p of activations to zero during training
      effect: Prevents co-adaptation of neurons; ensemble of thinned networks
      use_cases:
        - Fully connected layers in deep networks
        - When overfitting is observed
        - Standard in most deep learning architectures
      notes: "Typical rates: 0.2-0.5 for FC layers, 0.1-0.2 for conv layers; scale activations by 1/(1-p) at test time (or use inverted dropout)"
    - name: Batch Normalization
      mechanism: "Normalize activations per mini-batch: x̂ = (x - μ_batch) / √(σ²_batch + ε); y = γx̂ + β"
      effect: Reduces internal covariate shift; allows higher learning rates; slight regularization
      use_cases:
        - Deep CNNs
        - Very deep networks
        - When training is unstable
        - Enabling higher learning rates
      notes: Maintains running mean/variance for inference; can hurt performance with small batches or RNNs
    - name: Early Stopping
      mechanism: Monitor validation loss; stop training when val_loss stops improving for patience epochs
      effect: Prevents overfitting by stopping before model memorizes training data
      use_cases:
        - Any iterative training algorithm
        - When validation set available
        - Neural networks, gradient boosting
      notes: Save best model weights during training; restore after stopping; patience=5-20 typical
    - name: Data Augmentation
      mechanism: Apply label-preserving transformations to training data (rotation, flip, crop, color jitter for images)
      effect: Increases effective training set size; teaches invariances
      use_cases:
        - Computer vision (standard practice)
        - NLP (synonym replacement, back-translation)
        - Audio (pitch shift, time stretch)
        - When data is limited
      notes: Most effective regularization for vision; AutoAugment and RandAugment automate policy search
    - name: Label Smoothing
      mechanism: "Replace hard labels (0/1) with soft targets: y_smooth = (1 - ε) * y_hard + ε / K"
      effect: Prevents overconfidence; improves calibration; reduces gap between train and test accuracy
      use_cases:
        - Image classification (ResNet, EfficientNet)
        - Machine translation
        - When model is overconfident
        - Knowledge distillation teacher
      notes: Typical ε = 0.1; prevents model from assigning probability 1 to any class; improves generalization
    - name: Mixup / CutMix
      mechanism: "Mixup: interpolate pairs of images and labels. CutMix: cut and paste patches"
      effect: Linearizes training between examples; reduces memorization
      use_cases:
        - Image classification
        - When stronger regularization needed
        - State-of-the-art vision models
      notes: "Mixup: (x', y') = (λx_i + (1-λ)x_j, λy_i + (1-λ)y_j); CutMix often performs better by preserving local structure"