Phase 72.5.2.10 adds @v2/ab-test-analysis-pipeline, the experiment analysis
surface that turns hashed-account experiment telemetry into statistical
significance testing, effect size estimation, and automatic winner detection.
The pipeline composes @iris/analytics-ab through
buildV2ABTestAnalysisPipelineSurface. It accepts v2.experiment.exposed,
v2.experiment.converted, v2.experiment.engagement_recorded, and
v2.experiment.revenue_recorded events. Each event is keyed by accountIdHash,
experimentId, and variantId; raw account IDs are not accepted or emitted.
Policy#
- Policy ID:
v2-ab-test-analysis-pipeline-v1 - Source library:
@iris/analytics-ab - Required sections: statistical significance testing, effect size estimation, automatic winner detection
- Privacy: account-level joins use
accountIdHash; raw account IDs are never exposed
Output#
The surface emits:
- variant summaries with impressions, conversions, conversion rate, average engagement score, revenue per user, total revenue, and confidence intervals
- statistical-significance comparisons, including test method, p-value, confidence, and significance flag
- effect-size rows with control value, variant value, absolute lift, relative lift, and effect size
- automatic winner detection with rollout readiness and underperforming variants to pause
Gates#
check-v2-ab-test-analysis-pipeline.py validates the contract, service package,
source tokens, tests, docs, workflow wiring, Horde gates, and the Phase 72
checklist item. Required gates are ab-test-analysis-pipeline-service-present,
ab-test-analysis-pipeline-statistical-significance,
ab-test-analysis-pipeline-effect-size,
ab-test-analysis-pipeline-winner-detection, and
ab-test-analysis-pipeline-ci-wired.
Targeted verification:
pnpm --filter @v2/ab-test-analysis-pipeline run typecheck
pnpm --filter @v2/ab-test-analysis-pipeline run test
python V2/ue/Tools/check-v2-ab-test-analysis-pipeline.py