api: YAML JSON TOON Database
version: 1.0.0
format: toon
dataset:
  id: 2851
  slug: ml-training-algorithms
  title: Machine Learning Training Algorithms
  description: "Core ML training algorithms: gradient descent variants, backpropagation, optimization methods (SGD, Adam, RMSprop), and when to use each."
  summary: "Training algorithms for machine learning, including gradient descent in its batch, stochastic and mini-batch forms, momentum, RMSprop, Adam, backpropagation and learning-rate schedules, each with type, mechanism, update rule, pros, cons and when to use it, plus practical tips."
  use_when[2]: You need to understand or choose a training procedure.,You are debugging unstable training.
  caveats[2]: "Training behavior depends on initialization, batch size and learning rate; defaults do not transfer across all problems.",Backpropagation computes gradients; the optimizer decides how to use them.
  related[4]: deep-learning-optimizers,regularization-techniques,transfer-learning,ml-evaluation-metrics
  entry_count: 13
  fields[15]{section,field,type,example}:
    algorithms,name,string,Gradient Descent (Batch)
    algorithms,type,string,First-order optimization
    algorithms,how_it_works,string,"Computes gradient over entire training set, upd…"
    algorithms,update_rule,string,θ = θ - α × ∇J(θ)
    algorithms,pros,array,"[2 items]"
    algorithms,cons,array,"[2 items]"
    algorithms,use_when,string,"Small-to-medium datasets, convex optimization, …"
    algorithms,hyperparameters,object,"{γ (momentum),α (learning rate)}"
    algorithms,phases,array,"[3 items]"
    algorithms,key_formula,string,δ = ∇ₐL ⊙ σ'(z); ∇ₐL = Wᵀδ (for layer)
    algorithms,notes,string,Not an optimizer itself — computes gradients fo…
    algorithms,variants,array,"[3 items]"
    algorithms,strategies,object,"{step_decay,exponential_decay,1/t_decay,cosine_annealing}"
    algorithms,purpose,string,"Prevent oscillation near minimum, allow large i…"
    practical_tips,*,array,"[5 items]"
  category: AI & ML Terms
  category_slug: ai-ml-terms
  tags: "machine-learning,training,optimization,gradient-descent,backpropagation"
  view_count: 49
  created_at: 1777673262
  updated_at: 1777673262
data:
  algorithms[8]:
    - name: Gradient Descent (Batch)
      type: First-order optimization
      how_it_works: "Computes gradient over entire training set, updates parameters in opposite direction of gradient."
      update_rule: θ = θ - α × ∇J(θ)
      pros[2]: Converges to global minimum for convex functions,Stable convergence
      cons[2]: Slow for large datasets (full pass each iteration),Memory intensive
      use_when: "Small-to-medium datasets, convex optimization, need exact gradient"
    - name: Stochastic Gradient Descent (SGD)
      type: First-order optimization
      how_it_works: Updates parameters per training example; gradient estimate noisy but cheap.
      update_rule: "For each example: θ = θ - α × ∇J(θ; xᵢ, yᵢ)"
      pros[3]: Fast per update,Can escape shallow local minima (noise),Works with massive datasets
      cons[3]: Noisy convergence,May never converge exactly,Requires learning rate schedule
      use_when: "Large-scale learning, online learning, deep neural networks"
    - name: Mini-batch Gradient Descent
      type: First-order optimization
      how_it_works: Update per mini-batch (typically 32-512 examples); balance between SGD and batch.
      update_rule: "For each batch B: θ = θ - α × (1/|B|) Σ ∇J(θ; xᵢ, yᵢ)"
      pros[3]: More stable than SGD,Vectorized operations (GPU-friendly),Good convergence speed
      cons[2]: Need to tune batch size,Still some noise
      use_when: Default choice for most deep learning; works with GPU acceleration
    - name: Momentum
      type: Gradient descent with velocity
      how_it_works: Accumulates velocity in consistent gradient directions; dampens oscillations.
      update_rule: v = γv + α∇J(θ); θ = θ - v
      hyperparameters:
        "γ (momentum)": 0.9 typical
        "α (learning rate)": same as SGD
      pros[3]: Faster convergence on ravines,Reduces oscillations,Can escape shallow local minima
      cons[2]: May overshoot if momentum too high,Requires tuning γ
      use_when: "Training deep nets, ill-conditioned problems"
    - name: RMSprop
      type: Adaptive learning rate
      how_it_works: Divides learning rate by running average of squared gradients; adapt per-parameter.
      update_rule: "E[g²] = βE[g²] + (1-β)g²; θ = θ - α / √(E[g²] + ε) × g"
      hyperparameters:
        "β": 0.9 typical
        "α": 0.001 common
        "ε": 1e-8 numerical stability
      pros[3]: Adapts learning rate per parameter,Handles sparse gradients well,Works for non-convex
      cons[2]: May still need schedule,Biases early iterations
      use_when: "RNNs, sparse data, non-stationary objectives"
    - name: Adam (Adaptive Moment Estimation)
      type: Adaptive + momentum
      how_it_works: Combines momentum (first moment) and RMSprop (second moment) with bias correction.
      update_rule: m = β₁m + (1-β₁)g; v = β₂v + (1-β₂)g²; m̂ = m/(1-β₁ᵗ); v̂ = v/(1-β₂ᵗ); θ = θ - α × m̂ / (√v̂ + ε)
      hyperparameters:
        "β₁": 0.9 (momentum decay)
        "β₂": 0.999 (RMSprop decay)
        "α": 0.001 default
        "ε": "1e-8"
      pros[4]: Robust default,Fast convergence,Low memory,Works for most problems
      cons[2]: May generalize worse than SGD with momentum (some papers),Can diverge if α too large
      use_when: Default for most deep learning; quick prototyping
    - name: Backpropagation
      type: Gradient computation algorithm
      how_it_works: Applies chain rule efficiently through computational graph; forward pass then backward pass.
      phases[3]: "Forward: compute loss","Backward: compute ∂loss/∂params via chain rule","Update: optimizer step"
      key_formula: δ = ∇ₐL ⊙ σ'(z); ∇ₐL = Wᵀδ (for layer)
      notes: Not an optimizer itself — computes gradients for any optimizer to use. Works with any differentiable model.
      variants[3]: Standard backprop (DAG),Backprop through time (RNN),Reverse-mode autodiff (frameworks)
    - name: Learning Rate Schedules
      type: Meta-strategy
      strategies:
        step_decay: Drop LR by factor every N epochs
        exponential_decay: "α = α₀ × e^{-kt}"
        "1/t_decay": α = α₀ / (1 + kt)
        cosine_annealing: Cosine curve from α₀ to 0
        warmup: Gradually increase LR from 0 to α₀ in first few epochs
      purpose: "Prevent oscillation near minimum, allow large initial steps, fine-tune later"
      notes: "Adam often doesn't need schedule, but warmup helps early training stability"
  practical_tips[5]: Start with Adam (α=0.001) for quick results; switch to SGD+momentum (α=0.1) for final tuning if generalization suffers,"Normalize inputs (zero mean, unit variance) for stable gradient descent",Use gradient clipping (norm or value) to prevent exploding gradients (especially RNNs),Monitor training/validation loss curves to detect divergence or plateaus,Batch normalization helps stabilize gradients in deep networks