Skip to content

Holdouts & Incrementality

Holdouts and incrementality testing create persistent control groups, geo-holdouts, and switchback tests to measure the true incremental impact of changes.

Overview

Holdouts maintain control groups that never receive treatment, allowing you to measure the true incremental impact of changes by comparing treated users to untreated users.

Holdout Types

Persistent Holdouts

Maintain control groups over time:

{
  "holdout_type": "persistent",
  "control_percentage": 10,
  "assignment": "user_based",
  "persistence": "permanent"
}

Geo-Holdouts

Geographic control groups:

{
  "holdout_type": "geo",
  "control_regions": ["region_a", "region_b"],
  "treatment_regions": ["region_c", "region_d"]
}

Switchback Tests

Alternate between treatment and control:

{
  "holdout_type": "switchback",
  "pattern": "weekly",
  "control_weeks": [1, 3, 5],
  "treatment_weeks": [2, 4, 6]
}

Incrementality Measurement

Measure true incremental impact:

{
  "incrementality_measurement": {
    "control_group": "holdout_10_percent",
    "treatment_group": "full_treatment",
    "metrics": [
      "incremental_conversions",
      "incremental_revenue",
      "incremental_margin"
    ],
    "attribution_window_days": 30
  }
}

Best Practices

  1. Maintain holdouts: Keep control groups stable over time
  2. Measure incrementality: Focus on incremental impact, not absolute metrics
  3. Use geo-holdouts: When user-level holdouts aren't feasible
  4. Account for spillover: Consider cross-group effects
  5. Run long enough: Ensure sufficient data for measurement