Fighting Game · Guides & deep dives

V2 Game Mechanics A/B Testing

Primary package API:

4sections1 minread
Markdown source
On this page

This runbook covers TODOS.phase-72.72.12.1.6: a game mechanics A/B testing framework that splits players into deterministic groups, measures engagement and retention per variant, and reports statistical significance.

The service package is @v2/game-mechanics-ab-testing at apps/v2/game-mechanics-ab-testing/. Its contract is V2/ue/Content/V2/LiveOps/GameMechanicsABTesting_V2_Contract.json and the validation gate is V2/ue/Tools/check-v2-game-mechanics-ab-testing.py.

Framework Surface#

Primary package API:

  • assignV2GameMechanicsABVariant
  • buildV2GameMechanicsABTestingSurface

Experiment definitions include experiment ID, mechanic key, assignment salt, minimum sample size, confidence threshold, and weighted variants. Each variant has an ID, display name, weight, control marker, and mechanic config.

Measurement Flow#

  1. Assign players by hashed accountIdHash, experiment ID, and assignment salt.
  2. Emit v2.game_mechanic.exposed when a player sees the mechanic variant.
  3. Emit v2.game_mechanic.engagement_recorded with a non-negative engagement value.
  4. Emit v2.game_mechanic.retained when retention criteria are met.
  5. Build the surface to produce variant measurements and significance rows for engagement and retention.

The framework validates at least two variants, exactly one control, weights summing to 100, positive sample sizes, confidence thresholds between 0 and 1, supported event names, ISO timestamps, and non-negative engagement values.

Release Gates#

Required gates:

  • game-mechanics-ab-testing-package
  • game-mechanics-ab-testing-assignment
  • game-mechanics-ab-testing-measurement
  • game-mechanics-ab-testing-significance
  • game-mechanics-ab-testing-validation
  • game-mechanics-ab-testing-ci-wired

Verification#

bash
pnpm --filter @v2/game-mechanics-ab-testing test
pnpm --filter @v2/game-mechanics-ab-testing typecheck
python3 V2/ue/Tools/check-v2-game-mechanics-ab-testing.py
python3 V2/ue/Tools/check-v2-ci-workflow.py
python3 V2/tools/validate-v2-docs.py