api: YAML JSON TOON Database
version: 1.0.0
format: toon
dataset:
  id: 2791
  slug: experimental-design-pitfalls
  title: Experimental Design and Inference Pitfalls
  description: "Common threats to experiment validity, with a practical design adjustment and reporting consequence."
  summary: "Common threats to experiment validity, with a practical design adjustment and reporting consequence. Each row includes a practical application and a failure to check before using it."
  use_when[2]: Review a product experiment before collecting data.,Interpret a metric change without overstating causality.
  caveats[2]: Design guidance does not replace a task-specific statistical analysis.,Randomization reduces some biases but does not fix poor measurement.
  related[2]: probability-distribution-reference,statistics-basics
  reviewed_at: 2026-10-12
  sources[2]{title,url}:
    NIST design of experiments handbook,"https://www.itl.nist.gov/div898/handbook/pri/section3/pri3.htm"
    NIST statistical assumptions,"https://www.itl.nist.gov/div898/handbook/eda/section2/eda2.htm"
  reading[1]: experiment-checklist-before-trusting-a-metric-change
  provenance: "Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses."
  entry_count: 10
  fields[4]{section,field,type,example}:
    data,pitfall,string,Selection bias
    data,example,string,Only enthusiastic early users enter the experim…
    data,adjustment,string,Define the target population and recruitment pr…
    data,report,string,State who is outside the measured population.
  category: Mathematics
  category_slug: mathematics
  tags: "mathematics,experimental design pitfalls,reference,engineering"
  view_count: 70
  created_at: 1791798640
  updated_at: 1791798640
data[10]{pitfall,example,adjustment,report}:
  Selection bias,Only enthusiastic early users enter the experiment.,Define the target population and recruitment process.,State who is outside the measured population.
  Confounding,A new feature and a pricing change launch together.,Randomize assignment or separate the interventions.,Distinguish association from an identified causal effect.
  Peeking,Stop when a running p-value first falls below a threshold.,Use a planned fixed horizon or valid sequential method.,Describe the stopping rule.
  Multiple comparisons,Test many metrics and publish only the best.,Predefine primary outcomes and account for multiplicity.,Show the family of analyses performed.
  Low power,A small test misses a practically important effect.,Plan sample size around a meaningful effect and variance.,Report uncertainty rather than declaring no effect.
  Attrition,One group loses more participants before measurement.,Measure and investigate missingness by assignment.,Report dropout and sensitivity analyses.
  Unit mismatch,Randomize teams but analyze users as independent.,Use an analysis that respects the assignment unit.,State the number of independent units.
  Interference,Treated users change the experience of control users.,Consider cluster assignment or an interference model.,Describe spillover limitations.
  Metric gaming,A proxy improves while the real user outcome worsens.,Use guardrail metrics and inspect concrete outcomes.,Explain what the primary metric fails to capture.
  Post-treatment filtering,Remove participants based on behavior affected by treatment.,Define inclusion before assignment or justify the estimand.,Disclose exclusions and their timing.