This runbook covers TODOS.phase-72.72.12.1.1, TODOS.phase-72.72.12.1.2,
TODOS.phase-72.72.12.1.3, and TODOS.phase-72.72.12.1.7: remote configuration
for game parameters that live operations can change without a client patch,
config versioning with instant rollback when a bad change reaches production,
player segment targeting by region, platform, player level, and A/B test group,
and a config change audit log for full traceability. It also covers
TODOS.phase-72.72.12.1.8: validation rules that block obviously broken values
such as enemy health 0, drop rate 100%, and a decreasing XP curve before
publish. It also covers TODOS.phase-72.72.12.1.9: preview/staging support for
testing config changes in staging before production promotion. It also covers
TODOS.phase-72.72.12.1.10: delivery optimization with delta updates, client
caching, and minimal bandwidth usage. It also covers
TODOS.phase-72.72.12.1.13: maintenance mode that can redirect all clients to a
maintenance message during server updates with forced refresh, ETag
invalidation, short client staleness, and no client patch.
The service package is @v2/remote-config-service at
apps/v2/remote-config-service/. Its contract is
V2/ue/Content/V2/LiveOps/RemoteConfigService_V2_Contract.json and the
validation gate is V2/ue/Tools/check-v2-remote-config-service.py.
Game Parameters#
The initial service owns six server-tunable parameter families:
enemy.health.training-gruntfor enemy health.weapon.damage.iron-saberfor weapon damage.loot.drop-rate.legendary-shardfor reward drop rates.economy.price.skin-volt.softfor economy prices.economy.reward.daily-win.softfor economy rewards.progression.xp-curve.levels-1-5for XP curves.
Each parameter definition carries a default value, kind, description, unit,
minimum and maximum where applicable, and clientPatchRequired: false. Clients
can refresh these values without a client patch.
Primary package API:
buildV2RemoteConfigServiceSurfaceresolveV2RemoteConfigParameterupdateV2RemoteConfigParameterbuildV2RemoteConfigClientPayloadbuildV2RemoteConfigVersionHistoryplanV2RemoteConfigRollbackrollbackV2RemoteConfigVersionbuildV2RemoteConfigSegmentTargetingSurfaceresolveV2RemoteConfigForPlayerbuildV2RemoteConfigChangeAuditLogbuildV2RemoteConfigChangeAuditLogFromHistorybuildV2RemoteConfigValidationRulesSurfacebuildV2RemoteConfigPreviewStagingSurfacebuildV2RemoteConfigDeliveryOptimizationSurfacebuildV2RemoteConfigMaintenanceModeSurfaceresolveV2RemoteConfigMaintenanceRedirect
Delivery Flow#
- Live ops publishes a remote snapshot with a version ID, environment, generated timestamp, TTL, parameter definitions, and remote values.
- The service validates duplicate keys, unknown keys, numeric bounds, drop-rate caps, and non-decreasing XP curves.
- The client downloads
GET /v2/live-ops/remote-config/{environment}or revalidates withGET /v2/live-ops/remote-config/{environment}/etag/{etag}. - The client applies the
parametersmap frombuildV2RemoteConfigClientPayload. - A later operator change calls
POST /v2/live-ops/remote-config/{environment}/parameters/{key}and publishes a new snapshot version. No binary patch is required.
Versioning And Rollback Flow#
Every published config surface is recorded as an immutable version record with its version ID, generated timestamp, ETag, parameter hash, publisher, change summary, optional incident ID, and full snapshot. The active version is the only version clients receive from the normal config endpoint.
Operators can inspect version history through:
GET /v2/live-ops/remote-config/{environment}/versions
Rollback uses:
POST /v2/live-ops/remote-config/{environment}/rollback/{versionId}
The rollback planner validates that the target exists, is not the active
version, has not already been marked rolled back, and that the rollback version
ID is new. A successful rollback publishes a new active snapshot that restores
the selected previous parameter map and keeps requiresClientPatch: false. The
bad active version is marked rolled-back, while the restored snapshot has a
new version ID for auditability.
Config Change Audit Log#
Audit log endpoints are:
GET /v2/live-ops/remote-config/{environment}/audit-logGET /v2/live-ops/remote-config/{environment}/audit-log/{auditId}
buildV2RemoteConfigChangeAuditLogFromHistory derives immutable entries from
the version history. buildV2RemoteConfigChangeAuditLog accepts explicit audit
entries and validates duplicate audit IDs, environment mismatches, missing
reasons, and missing parameter diffs.
Each entry records:
changedBy, which captures who made the change.parameterChanges, which captures what changed with before/after values.changedAt, which captures when the change happened.reason, which captures why the change happened.
The normalized traceability block repeats who, what, when, and why for
operator review. Entries are immutable and include auditHash plus
previousAuditHash, forming a tamper-evident chain across initial publishes,
parameter updates, rollback publishes, and segment overrides. The focused gate
is check-v2-remote-config-audit-log.py.
Config Validation Rules#
Validation rule endpoints are:
GET /v2/live-ops/remote-config/{environment}/validation-rulesPOST /v2/live-ops/remote-config/{environment}/validate
buildV2RemoteConfigValidationRulesSurface evaluates proposed config changes
before publish and returns a report with blocksPublish, per-change issues,
rejectedValue, requestedBy, and reason. Required validation rules are:
known-parameter-keyenemy-health-positiveweapon-damage-non-negativedrop-rate-below-100-percenteconomy-price-non-negativeeconomy-reward-non-negativexp-curve-non-decreasingnumeric-bounds-enforced
The rules protect operators from obvious production-breaking values including
enemy health 0, drop rate 100%, negative economy prices, negative economy
rewards, and a decreasing XP curve. Any error issue sets blocksPublish: true.
The focused gate is check-v2-remote-config-validation-rules.py.
Config Preview And Staging#
Preview/staging endpoints are:
POST /v2/live-ops/remote-config/{environment}/previewPOST /v2/live-ops/remote-config/staging/promote
buildV2RemoteConfigPreviewStagingSurface starts from a production surface,
runs the validation rules, and builds a staging-only snapshot when the
validationReport is clean. Valid previews include stagingClientPayload,
previewDiffs, productionUnchanged: true, readyForProduction: true, and
requiresClientPatch: false.
Invalid previews keep production unchanged, preserve the production client
payload, omit staging payloads, and set readyForProduction: false so operators
cannot promote an unsafe change. The focused gate is
check-v2-remote-config-preview-staging.py.
Config Delivery Optimization#
Delivery optimization endpoints are:
GET /v2/live-ops/remote-config/{environment}/optimizedGET /v2/live-ops/remote-config/{environment}/delta/{etag}HEAD /v2/live-ops/remote-config/{environment}/etag/{etag}
buildV2RemoteConfigDeliveryOptimizationSurface compares the current client
payload against a previous client payload and selects cache-hit, delta, or
full delivery. The surface combines delta updates with client caching.
cache-hit means the client ETag is current and no payload download is needed.
delta sends only changed or removed parameters when that payload is smaller
than the full config. full remains the fallback when no previous payload
exists.
The cache policy exposes max-age, stale-while-revalidate, offline cache
seconds, If-None-Match revalidation, and a cache key. The surface reports
delta payload bytes, full payload bytes, estimated bytes saved,
compressionRecommended, minimalBandwidth: true, and
requiresClientPatch: false. The focused gate is
check-v2-remote-config-delivery-optimization.py.
Maintenance Mode#
Maintenance mode endpoints are:
GET /v2/live-ops/remote-config/{environment}/maintenance-modePOST /v2/live-ops/remote-config/{environment}/maintenance-mode
buildV2RemoteConfigMaintenanceModeSurface activates or clears maintenance mode
on top of the remote config service. Active maintenance publishes a new config
version, caps refreshTtlSeconds to the configured maxClientStalenessSeconds,
invalidates the previous ETag, and emits a forced refresh instruction with
invalidatedEtags, cacheBustToken, forceRefreshBeforeUse: true, and
propagationMode: 'force-refresh'.
The active client payload sets maintenanceActive: true,
redirectAllClients: true, and redirectRoute: '/maintenance'. Clients call
resolveV2RemoteConfigMaintenanceRedirect before entering gameplay; an active
payload returns a blocking redirect decision with the maintenance message, retry
delay, optional estimated resume time, optional status page URL, and optional
support URL. An inactive payload returns redirect: false, which clears the
maintenance route after the server update completes.
Operator flow:
- Open maintenance mode with a title, maintenance message, reason, estimated
resume time, status page URL, and
maxClientStalenessSeconds. - Clients refresh the maintenance endpoint, see the ETag change, and redirect
all clients to
/maintenancebefore gameplay or matchmaking entry. - During the update, the short client staleness window bounds how long a stale client can avoid the redirect.
- Close maintenance mode with
state: 'inactive'; clients force-refresh again and leave the redirect path with no client patch.
Required evidence is enforced by check-v2-remote-config-maintenance-mode.py.
Player Segment Targeting#
Segment targeting applies different remote config values by:
- region
- platform
- player level
- A/B test group
The targeting surface validates unique segment IDs, non-negative priorities, non-empty criteria, supported platforms, valid level ranges, known parameter keys, duplicate override keys, numeric bounds, drop-rate caps, and XP curve ordering. Targeting endpoints are:
GET /v2/live-ops/remote-config/{environment}/segmentsPOST /v2/live-ops/remote-config/{environment}/segments/evaluate
resolveV2RemoteConfigForPlayer evaluates the hashed player context against the
configured segments. Matched segments are ordered by descending priority with a
stable segment ID tie-breaker. A higher-priority segment wins when multiple
matched segments override the same parameter; unrelated parameters keep their
base remote config value. The returned segmented client payload includes matched
segment IDs, parameter source metadata, a segment-specific ETag, and
requiresClientPatch: false.
Release Gates#
Required gates:
remote-config-service-packageremote-config-service-game-parametersremote-config-service-no-client-patchremote-config-service-client-payloadremote-config-service-validationremote-config-service-ci-wiredremote-config-versioning-historyremote-config-versioning-rollback-planremote-config-versioning-instant-revertremote-config-versioning-client-payloadremote-config-versioning-ci-wiredremote-config-segment-targeting-surfaceremote-config-segment-targeting-dimensionsremote-config-segment-targeting-resolutionremote-config-segment-targeting-validationremote-config-segment-targeting-ci-wiredremote-config-audit-log-surfaceremote-config-audit-log-actorremote-config-audit-log-reasonremote-config-audit-log-diffremote-config-audit-log-hash-chainremote-config-audit-log-ci-wiredremote-config-validation-rules-surfaceremote-config-validation-rules-enemy-health-zeroremote-config-validation-rules-drop-rate-100-percentremote-config-validation-rules-xp-curveremote-config-validation-rules-unknown-keysremote-config-validation-rules-ci-wiredremote-config-preview-staging-surfaceremote-config-preview-staging-snapshotremote-config-preview-staging-production-unchangedremote-config-preview-staging-validation-gatedremote-config-preview-staging-promotion-readinessremote-config-preview-staging-ci-wiredremote-config-delivery-optimization-surfaceremote-config-delivery-optimization-deltaremote-config-delivery-optimization-cache-hitremote-config-delivery-optimization-cache-policyremote-config-delivery-optimization-minimal-bandwidthremote-config-delivery-optimization-ci-wiredremote-config-maintenance-mode-surfaceremote-config-maintenance-mode-redirectremote-config-maintenance-mode-messageremote-config-maintenance-mode-client-refreshremote-config-maintenance-mode-no-patchremote-config-maintenance-mode-ci-wired
Verification#
pnpm --filter @v2/remote-config-service test
pnpm --filter @v2/remote-config-service typecheck
python3 V2/ue/Tools/check-v2-remote-config-service.py
python3 V2/ue/Tools/check-v2-remote-config-versioning.py
python3 V2/ue/Tools/check-v2-remote-config-segment-targeting.py
python3 V2/ue/Tools/check-v2-remote-config-audit-log.py
python3 V2/ue/Tools/check-v2-remote-config-validation-rules.py
python3 V2/ue/Tools/check-v2-remote-config-preview-staging.py
python3 V2/ue/Tools/check-v2-remote-config-delivery-optimization.py
python3 V2/ue/Tools/check-v2-remote-config-maintenance-mode.py
python3 V2/ue/Tools/check-v2-ci-workflow.py
python3 V2/tools/validate-v2-docs.py