Machine Learning Training Algorithms
Core ML training algorithms: gradient descent variants, backpropagation, optimization methods (SGD, Adam, RMSprop), and when to use each.
Training algorithms for machine learning, including gradient descent in its batch, stochastic and mini-batch forms, momentum, RMSprop, Adam, backpropagation and learning-rate schedules, each with type, mechanism, update rule, pros, cons and when to use it, plus practical tips.
Use it when
- You need to understand or choose a training procedure.
- You are debugging unstable training.
Watch out for
- Training behavior depends on initialization, batch size and learning rate; defaults do not transfer across all problems.
- Backpropagation computes gradients; the optimizer decides how to use them.
The data
Algorithms
8 rows
| Name | Type | How it works | Update rule | Pros | Cons | Use when | Hyperparameters | Phases | Key formula | Notes | Variants | Strategies | Purpose |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gradient Descent (Batch) | First-order optimization | Computes gradient over entire training set, updates parameters in opposite direction of gradient. | θ = θ - α × ∇J(θ) |
|
| Small-to-medium datasets, convex optimization, need exact gradient | null | null | null | null | null | null | null |
| Stochastic Gradient Descent (SGD) | First-order optimization | Updates parameters per training example; gradient estimate noisy but cheap. | For each example: θ = θ - α × ∇J(θ; xᵢ, yᵢ) |
|
| Large-scale learning, online learning, deep neural networks | null | null | null | null | null | null | null |
| Mini-batch Gradient Descent | First-order optimization | Update per mini-batch (typically 32-512 examples); balance between SGD and batch. | For each batch B: θ = θ - α × (1/|B|) Σ ∇J(θ; xᵢ, yᵢ) |
|
| Default choice for most deep learning; works with GPU acceleration | null | null | null | null | null | null | null |
| Momentum | Gradient descent with velocity | Accumulates velocity in consistent gradient directions; dampens oscillations. | v = γv + α∇J(θ); θ = θ - v |
|
| Training deep nets, ill-conditioned problems |
| null | null | null | null | null | null |
| RMSprop | Adaptive learning rate | Divides learning rate by running average of squared gradients; adapt per-parameter. | E[g²] = βE[g²] + (1-β)g²; θ = θ - α / √(E[g²] + ε) × g |
|
| RNNs, sparse data, non-stationary objectives |
| null | null | null | null | null | null |
| Adam (Adaptive Moment Estimation) | Adaptive + momentum | Combines momentum (first moment) and RMSprop (second moment) with bias correction. | m = β₁m + (1-β₁)g; v = β₂v + (1-β₂)g²; m̂ = m/(1-β₁ᵗ); v̂ = v/(1-β₂ᵗ); θ = θ - α × m̂ / (√v̂ + ε) |
|
| Default for most deep learning; quick prototyping |
| null | null | null | null | null | null |
| Backpropagation | Gradient computation algorithm | Applies chain rule efficiently through computational graph; forward pass then backward pass. | null | null | null | null | null |
| δ = ∇ₐL ⊙ σ'(z); ∇ₐL = Wᵀδ (for layer) | Not an optimizer itself — computes gradients for any optimizer to use. Works with any differentiable model. |
| null | null |
| Learning Rate Schedules | Meta-strategy | null | null | null | null | null | null | null | null | Adam often doesn't need schedule, but warmup helps early training stability | null |
| Prevent oscillation near minimum, allow large initial steps, fine-tune later |
Practical tips
- Start with Adam (α=0.001) for quick results; switch to SGD+momentum (α=0.1) for final tuning if generalization suffers
- Normalize inputs (zero mean, unit variance) for stable gradient descent
- Use gradient clipping (norm or value) to prevent exploding gradients (especially RNNs)
- Monitor training/validation loss curves to detect divergence or plateaus
- Batch normalization helps stabilize gradients in deep networks
Fields
The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.
| Section | Field | Type | Example |
|---|---|---|---|
| algorithms | name | string | Gradient Descent (Batch) |
| algorithms | type | string | First-order optimization |
| algorithms | how_it_works | string | Computes gradient over entire training set, upd… |
| algorithms | update_rule | string | θ = θ - α × ∇J(θ) |
| algorithms | pros | array | [2 items] |
| algorithms | cons | array | [2 items] |
| algorithms | use_when | string | Small-to-medium datasets, convex optimization, … |
| algorithms | hyperparameters | object | {γ (momentum),α (learning rate)} |
| algorithms | phases | array | [3 items] |
| algorithms | key_formula | string | δ = ∇ₐL ⊙ σ'(z); ∇ₐL = Wᵀδ (for layer) |
| algorithms | notes | string | Not an optimizer itself — computes gradients fo… |
| algorithms | variants | array | [3 items] |
| algorithms | strategies | object | {step_decay,exponential_decay,1/t_decay,cosine_annealing} |
| algorithms | purpose | string | Prevent oscillation near minimum, allow large i… |
| practical_tips | * | array | [5 items] |
Fetch the same bytes
The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.
curl "https://yjtoon.yunusemrevurgun.com/api/dataset/ml-training-algorithms?format=toon"
const res = await fetch( "https://yjtoon.yunusemrevurgun.com/static-data/dataset/ml-training-algorithms.toon" ); const toon = await res.text();
Rate limit: 120 requests per minute per IP, no key and no signup. API reference →