# V2 Dedicated Server Operations Runbook

This runbook is the operational contract for deploying, monitoring, and
troubleshooting `V2Server` dedicated server builds. It covers
`TODOS.phase-72.72.2.1.15` and ties together the server build, health, budget,
logging, replay, hot-patch, integration, and determinism gates from phase 72.

## Deployment

### Artifacts

Server artifacts are produced by the BuildGraph server binary compilation
pipeline:

- `LinuxServer-x86_64`
- `LinuxServer-ARM64`
- `Win64Server`

Each artifact is built from target `V2Server` in `Shipping` configuration with
server-only cooking, pak archives, deterministic floating-point settings, and no
client payload. Promotion requires the binary contract
`V2/ue/Content/V2/Netcode/Server/ServerBinaryCompilationPipeline_V2_Contract.json`.

### Preflight

Run these gates before any artifact can receive player traffic:

```bash
python3 V2/ue/Tools/check-v2-server-binary-compilation-pipeline.py
python3 V2/ue/Tools/check-v2-server-tick-rate-config.py
python3 V2/ue/Tools/check-v2-server-health-monitoring.py
python3 V2/ue/Tools/check-v2-server-budget-enforcement.py
python3 V2/ue/Tools/check-v2-server-log-aggregation.py
python3 V2/ue/Tools/check-v2-server-replay-recording.py
python3 V2/ue/Tools/check-v2-server-hot-patching.py
python3 V2/ue/Tools/check-v2-server-integration-test-harness.py
python3 V2/ue/Tools/check-v2-server-determinism-verification.py
```

The release owner records the artifact id, Git SHA, platform id, config
revision, region, and rollback artifact id in the deployment ticket before
opening matchmaking.

### Boot Contract

Dedicated server launch must include a region id, match id, supported tick rate,
and headless rendering:

```bash
V2Server /Game/Maps/L_OnlineHeadless -server -log -unattended -nullrhi \
  -NoSound -v2region=us-east -v2matchid=<match-id> -v2tickrate=60
```

Supported tick rates are `30`, `60`, and `128`. Default production boot is
`60 Hz`; `128 Hz` requires an explicit competitive-mode allocation, and `30 Hz`
is the recovery profile for overloaded or low-population sessions.

### Rollout

Roll out new server binaries by canarying new allocations first:

- Start at 5 percent of new sessions for one region.
- Block expansion while `bServerHealthEligibleForNewSessions` is false for any
  canary host.
- Drain old hosts with new-session admission disabled, then use graceful player
  migration for active sessions.
- Promote only after health, budget, replay, log aggregation, integration
  harness, and determinism checks are green for the canary artifact.
- Roll back by disabling new allocations for the candidate artifact, draining
  live sessions, and restoring the previous `V2Server` artifact id.

## Monitoring

### Health Dashboard

The live dashboard must display the health state from `FV2ServerHealthReport`,
including these structured signals:

- `ServerHealth.TickOverrun`
- `ServerHealth.MemoryHigh`
- `ServerHealth.PlayerCap`
- `ServerHealth.CrashDetected`
- `ServerHealth.CrashHeartbeatStale`

Default thresholds are `20 ms` tick overrun, `3` consecutive tick overruns,
`8192 MiB` memory usage, `128` players, and `15 seconds` maximum crash heartbeat
age. Any degraded or crash-suspected host is ineligible for new sessions.

### Budget Dashboard

Budget enforcement publishes CPU, memory, garbage collection, drain, and kill
signals:

- `ServerBudget.CpuOverrun`
- `ServerBudget.MemorySoftLimit`
- `ServerBudget.MemoryHardLimit`
- `ServerBudget.AggressiveGarbageCollection`
- `ServerBudget.NewSessionDrain`
- `ServerBudget.RunawayServerKill`
- CPU utilization target: at or below `90 percent`
- game thread target: at or below `16.6 ms`
- server tick target: at or below `20 ms`
- aggressive garbage collection: starts at `7168 MiB`
- soft memory limit: `8192 MiB`
- hard memory kill threshold: `12288 MiB`
- process exit threshold: `10` consecutive CPU overruns

When the policy requests `DrainNewSessions`, matchmaking must stop placing new
players on the host. When the policy requests `TerminateRunawayServer`, crash
diagnostics and the final structured event batch must be preserved first.

### Logs, Replays, And Profiles

Structured server events publish only through the event bus topics
`v2.match.server.lifecycle`, `v2.match.server.player-session`,
`v2.match.server.authoritative-state`, and `v2.match.server.admin-command`. Raw
log lines stay out of aggregation.

Replay recording must capture all authoritative inputs needed for dispute
resolution, regression, and determinism reproduction. Performance profiling mode
is enabled only for investigation windows and must keep rollback state isolated
from profiling and telemetry side effects.

### Alerts

Page server on-call when any of these conditions persist for two consecutive
samples:

- crash heartbeat stale or crash detected
- memory above `8192 MiB`
- CPU above `90 percent`
- tick overrun count at or above `3`
- player count at or above `128`
- log aggregation shutdown flush failure
- determinism final state hash mismatch

## Troubleshooting

### Host Fails Preflight

Rerun the preflight gate that failed, inspect the associated contract JSON, and
compare the artifact id to the deployment ticket. Common causes are a stale
server binary, unsupported tick rate, unsigned hot-patch manifest, or a platform
artifact that was built for the wrong architecture.

### New Sessions Stop Landing

Check `bServerHealthEligibleForNewSessions`, `bServerBudgetNewSessionsDrained`,
and `ServerHealth.PlayerCap`. If the host is healthy but drained, clear the
drain only after CPU, memory, player count, and crash heartbeat signals are
green.

### Tick Or CPU Overruns

Confirm the current tick rate and inspect `ServerHealth.TickOverrun`,
`ServerBudget.CpuOverrun`, and the performance profiling capture. For overloads,
move new allocations away from the host, use the `30 Hz` recovery profile for
eligible sessions, and preserve the profiler artifact before terminating a
runaway process.

### Memory Pressure

If memory crosses `7168 MiB`, confirm aggressive garbage collection was
requested. If it crosses `8192 MiB`, drain new sessions. If it reaches
`12288 MiB`, preserve crash diagnostics, publish the final structured batch, and
let the budget policy terminate the server.

### Crash Heartbeat Failure

Crash heartbeat age above `15 seconds` moves the host to crash-suspected state.
Block new sessions, collect crash diagnostics, keep replay artifacts attached to
the match id, and migrate active players through the graceful shutdown flow when
the process still responds.

### Hot Patch Rejected

Verify the manifest signature, monotonic revision, payload size below `65536`
bytes, and config domain. Runtime hot patches are limited to
`ServerConfig.BalanceTweaks` and `ServerConfig.FeatureFlags`; changes requiring
a restart must ship through a new binary rollout.

### State Divergence

Collect the replay id, input stream hash, final state hash, server artifact id,
map path, tick rate, and match seed. Reproduce with the server integration test
harness first, then run server determinism verification with the same input
stream. A final state hash mismatch blocks promotion until the rollback or
authoritative-state change is fixed.

### Log Aggregation Backlog

Confirm event bus connectivity and the aggregation batch limits: `128` events,
`262144` bytes, and `2 seconds` flush interval. During shutdown, the final flush
must complete before the host exits or migrates players.

## Verification

The static contract lives at
`V2/ue/Content/V2/Netcode/Server/ServerOperationsRunbook_V2_Contract.json`.
Repository validation is enforced by
`V2/ue/Tools/check-v2-server-operations-runbook.py`, plus the phase 72 server
checkers listed in the preflight section.