Transfer Learning and Fine-Tuning

Transfer learning strategies: feature extraction, fine-tuning, domain adaptation, multi-task learning, prompt tuning, LoRA, and pretraining paradigms.

AI & ML Terms
8 entries
CC0 — public domain

JSON10,174 B
YAML7,066 B · −31%
TOON6,177 B · −39%

Strategies for adapting pretrained models: feature extraction, full fine-tuning, layer-wise learning-rate decay, progressive unfreezing, domain adaptation, LoRA, prompt and prefix tuning and multi-task learning, with approach, when to use, pros, cons and examples.

Use it when

You have limited data or compute and want to start from a pretrained model.
You are choosing how much of a model to update.

Watch out for

Pretrained models carry biases and limitations from their original data.
More parameter updates give more flexibility and more risk of forgetting or overfitting.

The data

Strategies

Name	Approach	When to use	Pros	Cons	Example	Notes
Feature Extraction (Frozen Backbone)	Use pretrained model as fixed feature extractor; train only new classification head	Small target dataset Target domain similar to source Limited compute resources	Fast training No catastrophic forgetting Minimal overfitting risk	Limited adaptation to target domain Features may not be optimal for new task	ResNet-50 pretrained on ImageNet → freeze all layers → train only final FC layer on custom dataset	null
Fine-Tuning (Full Model)	Unfreeze all layers; continue training entire model on target data with lower learning rate	Large target dataset Target domain differs from source Maximum performance needed	Full adaptation to target domain Best possible performance	Risk of catastrophic forgetting Requires more data Computationally expensive	BERT pretrained → fine-tune all 110M parameters on SQuAD QA dataset with LR=2e-5	null
Layer-wise Learning Rate Decay	Assign lower LR to earlier layers, higher LR to later layers during fine-tuning	Medium-sized target dataset Want to preserve generic low-level features	Balances preservation and adaptation Reduces forgetting of early features	Adds hyperparameter (decay rate) Requires tuning	LR(layer_i) = base_lr × decay^(depth - i); typical decay=0.9-0.95	null
Progressive Unfreezing	Start with frozen backbone + trainable head; gradually unfreeze layers from top to bottom	Small target dataset Want controlled adaptation Avoid catastrophic forgetting	Gradual adaptation reduces shock to pretrained weights Better stability	Slower training process More complex training loop	Phase 1: train head only. Phase 2: unfreeze last block + head. Phase 3: unfreeze all.	null
Domain Adaptation	Align feature distributions between source and target domains (e.g., adversarial training, MMD loss)	Source and target domains differ significantly Labeled target data scarce Domain shift is known issue	Explicitly addresses domain gap Works with unlabeled target data	More complex training Requires domain-specific techniques	DANN (Domain-Adversarial Neural Network): gradient reversal layer makes features domain-invariant	null
LoRA (Low-Rank Adaptation)	Inject trainable low-rank decomposition matrices into attention/FFN layers; freeze pretrained weights	Fine-tuning large models (LLMs, ViT) Memory-constrained environments Multiple downstream tasks from same base model	Dramatically fewer trainable parameters (0.1-1% of original) No inference latency overhead (merged into weights) Multiple LoRA adapters can share same base model	Slight accuracy drop vs full fine-tuning Rank hyperparameter to tune	LLaMA 7B: full fine-tuning = 7B params; LoRA (r=8) = 4.7M params (0.07%)	Used for most LLM fine-tuning (Alpaca, Vicuna, etc.)
Prompt Tuning / Prefix Tuning	Learn soft prompts (continuous vectors) prepended to input; freeze entire model	Very large models where even LoRA is expensive Multiple tasks from single model When prompt engineering is natural fit	Minimal trainable parameters (thousands vs billions) No model modification needed Easy to switch between tasks	Lower performance than full fine-tuning Only works well for very large models (10B+)	T5-11B: learn 100-token soft prompt (128-dim each) = 1.28M params vs 11B full model	null
Multi-Task Learning	Train single model on multiple related tasks simultaneously with shared backbone	Multiple related tasks available Data-scarce individual tasks Want shared representations	Improved generalization via inductive bias Single model for multiple tasks Data augmentation across tasks	Task interference possible Complex loss weighting Harder to debug	BERT pretraining: MLM + NSP simultaneously. MT-DNN: shared BERT + task-specific heads.	null

Fields

The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.

Section	Field	Type	Example
strategies	`name`	string	Feature Extraction (Frozen Backbone)
strategies	`approach`	string	Use pretrained model as fixed feature extractor…
strategies	`when_to_use`	array	[3 items]
strategies	`pros`	array	[3 items]
strategies	`cons`	array	[2 items]
strategies	`example`	string	ResNet-50 pretrained on ImageNet → freeze all l…
strategies	`notes`	string	Used for most LLM fine-tuning (Alpaca, Vicuna, …

Fetch the same bytes

The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.

curl "https://yjtoon.yunusemrevurgun.com/api/dataset/transfer-learning?format=toon"

const res = await fetch(
  "https://yjtoon.yunusemrevurgun.com/static-data/dataset/transfer-learning.toon"
);
const toon = await res.text();

Rate limit: 120 requests per minute per IP, no key and no signup. API reference →