Disciplines · Runbooks

Runbook: V1 Observability, Alert Response, and Supervised Game Day

With OSHUN_OBSERVABILITY_ENABLED=1, each environment starts a private Compose profile containing Prometheus, Alertmanager, Grafana, black-box probes, PostgreSQL and Redis exporters, and a textfile exporter for backup state.

12sections7 minread
Markdown source
On this page

Owner: production operations duty officer (rota:prod-ops). Last reviewed: 2026-09-15. Version: 1.

Use this runbook when a V1 Hetzner alert fires, a dashboard signal turns red, the external dead-man signal stops, or a supervised Task 13.7 game day is scheduled. It covers the deployed source path in infra/hetzner, not the larger development stack in docker/docker-compose.yml.

1. What is deployed#

With OSHUN_OBSERVABILITY_ENABLED=1, each environment starts a private Compose profile containing Prometheus, Alertmanager, Grafana, black-box probes, PostgreSQL and Redis exporters, and a textfile exporter for backup state. Prometheus and Alertmanager are the evaluator/router; Grafana is the operator workspace. None joins oshun-edge, and all three operator ports bind only to 127.0.0.1.

Two independent external endpoints and receiver failure domains are mandatory:

  • alert-webhook-url receives firing and resolved incident payloads.
  • deadman-webhook-url receives the always-firing OshunDeadMansSwitch every minute and must alert from outside Hetzner when that signal stops.

Both URL files and grafana-admin-password live beside the stack .env, are owned by the deploy user, share its private group, and have mode 0640. Only Alertmanager/Grafana receive that supplemental numeric group, allowing their non-root processes to read the bind-mounted files without exposing values in container environment or inspect output. The deploy script rejects missing, same-platform, identical, empty, symlinked, or incorrectly permissioned files. The endpoint values and password are never committed. Evidence admission also rejects incident and dead-man receipts that name the same receiver failure domain.

This split is deliberate. Prometheus detects a frozen service while the host is alive; the dead-man receiver detects the loss of Prometheus, Alertmanager, or the whole host. A local green dashboard is not proof that the external path is working.

2. Enablement and access#

Provision the three private files, then set:

dotenv
OSHUN_OBSERVABILITY_ENABLED=1
PROMETHEUS_LOOPBACK_PORT=9090
ALERTMANAGER_LOOPBACK_PORT=9093
GRAFANA_LOOPBACK_PORT=3000

Production refuses deployment when observability is disabled. On a 16 GiB CCX23, enable the profile for only one co-hosted environment; two observed stacks require separate boxes or CCX33+. Deploy normally so the health gate includes the profile.

Reach the private consoles through an operator SSH tunnel:

bash
ssh -N \
  -L 3000:127.0.0.1:3000 \
  -L 9090:127.0.0.1:9090 \
  -L 9093:127.0.0.1:9093 \
  deploy@<host>

Open Grafana at http://127.0.0.1:3000 and select Oshun V1 Operations. Its first row answers four questions only: are critical alerts firing, is the BFF available, is request latency inside the boundary, and is the off-box recovery point fresh. The service selector narrows the reachability timeline. Every alert links back to the matching section below.

3. First five minutes of any alert#

  1. Acknowledge the external notification and record its notification id and receipt time. Never paste its URL, token, headers, message body containing actor data, or screenshots with secrets into evidence.

  2. Open the Grafana dashboard through the tunnel. Record the stack, alert name, affected service, first firing time, and dashboard time range.

  3. Confirm the deployed source before changing anything:

    bash
    tr -d '\r\n' </opt/oshun/bundle/source-commit
docker compose --profile observability -p oshun-<stack> \
  --env-file /opt/oshun/<stack>/.env \
  -f /opt/oshun/bundle/docker-compose.yml ps

  4. Declare the incident through the external channel. Name the production operations duty officer, the incident/ticket id, affected actor capability, and the next update time.

  5. Choose the narrow section below. Do not restart the whole stack to erase the symptom before preserving alert, log, source, and container-state evidence.

BFF or public API unavailable#

Alert: OshunBffUnavailable (critical after 30 seconds).

  1. Compare probe_success{job="blackbox-http",service="bff"} with the BFF container state. A running or paused container with a zero answer probe is a process/wedge problem; an absent container is a rollout/process-loss problem.

  2. Inspect the last 200 BFF log lines and its restart/OOM state. Do not include bearer tokens, prompts, actor content, or complete environment dumps in the incident record.

  3. If the current immutable image introduced the fault, roll back only BFF:

    bash
    /opt/oshun/bundle/scripts/deploy.sh <stack> <last-known-good-sha-tag> bff

  4. If the process is deliberately paused by a game day, use the recorded reversal or v1-supervised-game-day.mjs --recover-state <state.json>. Never create a second replacement container while the paused writer remains.

  5. Verify container health, answer-probe recovery, resolved Alertmanager state, https://api.<domain>/health, and the unchanged/rolled-back source identity.

Data dependency unavailable#

Alert: OshunDependencyUnavailable.

The PostgreSQL and Redis exporters execute real queries, so they detect a frozen process rather than accepting a TCP handshake as health. Qdrant and MinIO are checked through answer-capable HTTP readiness endpoints.

Surface unavailable#

Alert: OshunSurfaceUnavailable. Confirm whether one surface, every surface, or the Caddy edge is affected. One surface may be rolled back by immutable image tag. If all public hosts fail while internal probes remain green, inspect Caddy and DNS/TLS. If internal and public probes fail together, follow the dependency or regional-loss runbook rather than restarting Caddy.

BFF errors or latency#

Alerts: OshunBffHighErrorRate, OshunBffHighLatency.

Filter the dashboard to the firing interval and inspect route-labelled error rates. Preserve a sanitized trace/request id, not request content. Check provider, database, Redis, and Qdrant panels before assigning the symptom to the BFF. Roll back only when the onset aligns with the deployed source and the prior immutable image is known good.

Backup stale or failed#

Alerts: OshunLocalBackupStale, OshunOffboxBackupUnhealthy, OshunOffboxBackupStale.

  1. Read the backup metric and the latest status/receipt; do not infer freshness from a running container.
  2. Follow v1-disaster-recovery.md and infra/hetzner/README.md. A local dump is not a box-loss recovery point.
  3. Do not run a destructive game day while either local or off-box backup state is absent, stale, disabled, or failed.

Alert delivery failing#

Alert: OshunAlertDeliveryFailing. Alertmanager is recording a failed external notification, so the same receiver may not deliver this alert. Check the independent dead-man system first, then inspect Alertmanager metrics/logs through the tunnel. Rotate or repair the receiver file outside Git and redeploy. Do not paste the URL into a ticket or command line.

Monitoring target down#

Alert: OshunMonitoringTargetDown. Determine whether the target process is down or only its exporter/scrape path. A healthy target with a failed exporter is telemetry blindness and remains an incident; do not silence it as harmless.

Dead-man signal missing#

This alert is raised by the external watchdog, not by the lost stack. Treat it as potential regional loss until an operator proves otherwise. If the box is gone, follow v1-disaster-recovery.md. If the box is reachable, check Prometheus, Alertmanager, outbound DNS/TLS, and the distinct dead-man endpoint. A current local OshunDeadMansSwitch firing series does not prove the external receiver saw it.

4. Supervised Task 13.7 game day#

Preconditions#

  • Use a production-like staging environment with no real actors. Production additionally requires explicit production confirmation and change approval.
  • The operator and approver are distinct stable identities.
  • The exact candidate is deployed and appears in /opt/oshun/bundle/source-commit.
  • All monitoring containers are healthy; local and off-box backup metrics are successful and no older than 25 hours.
  • The external watchdog supplies a sanitized receipt from the preceding 24 hours proving current dead-man delivery and a successful missing-signal page test from the preceding 30 days.
  • Set OSHUN_GAME_DAY_ALLOWED=true, redeploy BFF so its enforced container label changes, announce the window, and prepare the external alert receipt path.

Execute#

From the Hetzner host, run:

bash
node /opt/oshun/bundle/scripts/v1-supervised-game-day.mjs \
  --stack staging \
  --candidate <40-hex-source-commit> \
  --run-id gd-<UTC-date>-<sequence> \
  --operator operator:<name> \
  --approver operator:<different-name> \
  --ticket incident:<id> \
  --deadman-receipt /opt/oshun/staging/game-day/deadman-receipt.json \
  --external-receipt /opt/oshun/staging/game-day/external-alert-receipt.json \
  --state /opt/oshun/staging/game-day/run.state.json \
  --output /opt/oshun/staging/game-day/run.json \
  --confirm NO_REAL_ACTORS_AND_ROLLBACK_APPROVED

The deploy workflow installs that exact committed script in the bundle. The runner:

  1. preflights source identity, opt-in label, the provisioned dashboard and live Prometheus datasource, monitoring, backup freshness, and the dead-man receipt;
  2. atomically records the reversal before freezing BFF;
  3. waits for the answer probe, Prometheus alert, and Alertmanager alert;
  4. waits up to five minutes for the supervisor to write the external receipt;
  5. validates human triage and the rollback decision;
  6. unpauses BFF in finally, then verifies health, probe recovery, alert resolution, public health, and source identity.

The dead-man receipt is written only after checking the external watchdog's own event history and its missing-signal test:

json
{
  "schemaVersion": "oshun.v1-deadman-receipt.v1",
  "stack": "staging",
  "notificationId": "deadman:<sanitized-id>",
  "receiverFailureDomain": "watchdog.example.org",
  "observedAt": "<RFC3339>",
  "lastSignalAt": "<RFC3339-within-five-minutes-of-observedAt>",
  "missingSignalAlertTested": true,
  "missingSignalAlertTestedAt": "<RFC3339-within-30-days>",
  "testOutcome": "passed",
  "sanitized": true
}

Create the external receipt only after reading the external system:

json
{
  "schemaVersion": "oshun.v1-external-alert-receipt.v1",
  "runId": "gd-<UTC-date>-<sequence>",
  "stack": "staging",
  "alertName": "OshunBffUnavailable",
  "status": "firing",
  "notificationId": "notification:<sanitized-id>",
  "receiverFailureDomain": "incident-service.example.net",
  "receivedAt": "<RFC3339>",
  "triagedAt": "<RFC3339>",
  "triagedBy": "operator:<name>",
  "diagnosis": "BFF answer probe failed after the supervised process freeze",
  "decision": "rollback",
  "findings": [],
  "noUnrecordedFailures": true,
  "sanitized": true
}

If the runner is killed with SIGKILL, a second process must run:

bash
node /opt/oshun/bundle/scripts/v1-supervised-game-day.mjs \
  --recover-state /opt/oshun/staging/game-day/run.state.json

Then finish public health, alert-resolution, and source checks manually. An emergency reversal is recovery work, not a passing game day; schedule a fresh run only after the cause is repaired.

Findings and evidence#

Every observed failure must carry a unique id, a followUpRef, and an evalCaseId from docs/audits/eve-observability-game-day/task-13-7-eval-family.json. If nothing matches, add gd-uncatalogued-finding temporarily, create a precise new eval case and follow-up, and keep the game day failed until retest. A record with any finding cannot pass admission.

Retain only:

  • the runner output and final state file;
  • the two sanitized external receipts and their SHA-256 bindings;
  • exported dashboard/alert screenshots with secrets and actor data removed;
  • the immutable source commit and container image digest;
  • follow-up/eval updates for every finding.

Verify the retained record independently:

bash
node scripts/operations/v1-supervised-game-day.mjs \
  --verify-record docs/audits/eve-observability-game-day/<date>.json \
  --external-receipt docs/audits/eve-observability-game-day/<date>.external.json \
  --deadman-receipt docs/audits/eve-observability-game-day/<date>.deadman.json

Task 13.7 remains open until the real run passes, both external paths are demonstrated, the evidence is admitted, and independent confirmatory and adversarial reviews bind to the exact candidate and record.