Pinned excerpts and command output behind the chapter's slides (guide
stack-operations) and its brief (Eve document
cecd95f3-4e08-4157-a69b-adc3453c91da). Every excerpt is the repository at
commit b6f1ccf1c111a1b364faa114524aad0181975ed5, copied by the command shown above it. No deployment, rollback,
drill or evaluation ran. One thing was looked at on a real machine, the dev
server's compose status, and it is at the end.
Values read from the code, configuration and pages#
Produced by check-data.py in
docs/audits/presentation-center-eve-redesign-stack-operations/2026-10-07/,
which reads each value from the pinned source and compares it with the words on
the slides. All twenty comparisons were equal. The values it read:
{
"hetznerImages": {
"postgres": "pgvector/pgvector:pg16",
"redis": "valkey/valkey:8.1.10-alpine@sha256:d2e18f3410b6f616de1417f570fa55261af2898b9c5b2cfb6781ce2373ea43d1",
"qdrant": "qdrant/qdrant:v1.19.1",
"minio": "quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
"minio-init": "minio/mc:latest"
},
"devImages": {
"postgres": "pgvector/pgvector:pg16",
"redis": "valkey/valkey:8.1.10-alpine@sha256:d2e18f3410b6f616de1417f570fa55261af2898b9c5b2cfb6781ce2373ea43d1",
"minio": "quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
"minio-init": "minio/mc:latest"
},
"hetznerServices": [
"postgres",
"redis",
"qdrant",
"minio",
"minio-init",
"pg-backup",
"offbox-backup",
"bff",
"content-service",
"metis-api-gateway",
"metis-worker",
"web",
"admin",
"tenant-admin",
"telegram-miniapp",
"metis-web",
"metis-admin",
"psyche-api-gateway",
"psyche-orchestrator",
"psyche-behavior-engine",
"psyche-perception-engine",
"psyche-computer-use",
"psyche-conferencing",
"psyche-knowledge-base",
"psyche-persona-service",
"psyche-avatar-engine",
"psyche-voice-engine",
"alertmanager",
"blackbox-exporter",
"postgres-exporter",
"redis-exporter",
"backup-metrics",
"prometheus",
"grafana"
],
"bffWaitsForHealthy": [
"postgres",
"redis",
"qdrant",
"minio"
],
"valkeyComment": "Valkey (BSD-3), not Redis 7.4 (RSALv2 or SSPLv1) — tracker T.01.08. The service keeps the name `redis` so every REDIS_URL below still resolves. `valkeydata` is a NEW volume: Valkey cannot read Redis 7.4's RDB v12, and the owner chose to flush at cutover (2026-09-15). What is lost is the BullMQ queue's in-flight jobs and the day's cost-ledger counters; drain the queue before deploying this. `redisdata` stays for rollback.",
"valkeyEvictionPolicy": "volatile-lru",
"qdrantComment": true,
"minioComment": true,
"hetznerHeaderStillSaysRedis7": true,
"observabilityIsAProfile": true,
"deploySteps": [
[
"1",
"docker login to GHCR (when a token is provided)"
],
[
"2",
"Sync edge config (Caddyfile + sites + compose) bundle → /opt/oshun/edge,"
],
[
"3",
"Pull the target images"
],
[
"4",
"Bring up the data tier and wait for it to be healthy"
],
[
"5",
"Apply the ordered Oshun Prisma migration history via a one-off container"
],
[
"6",
"Roll the application services"
],
[
"7",
"Wait for every target container to be healthy; on failure, dump the"
],
[
"8",
"Public smoke through the edge (api/metis-api/psyche-api /health + apex)"
],
[
"9",
"Prune dangling images"
]
],
"deployRollbackForm": true,
"deployStopsOnBadMigration": true,
"deployHasNoJobStep": true,
"workflowFlow": "plan (resolve env + affected services) → build (one GHCR image per service) → [production approval] → deploy (rsync bundle + run infra/hetzner/scripts/deploy.sh over SSH) → public smoke.",
"workflowHasProductionApproval": true,
"launchLocales": [
"en-US",
"es-US",
"fr-FR",
"de-DE",
"ar",
"he",
"ja-JP",
"pt-BR"
],
"rtlLocales": [
"ar",
"he"
],
"criticalPrefixes": [
"assistant.disclosure.",
"evidence.",
"auth.minimum_version_required",
"privacy."
],
"fallbackOrderOnPage": true,
"budgetPaths": [
"/*",
"/search",
"/domains/nisaba"
],
"defaultTimings": {
"first-contentful-paint": 1800,
"largest-contentful-paint": 2500,
"max-potential-fid": 200,
"cumulative-layout-shift": 0.1,
"total-blocking-time": 250,
"speed-index": 3000
},
"timingsPerEntry": [
6
],
"metricsNamedAnywhere": [
"cumulative-layout-shift",
"first-contentful-paint",
"largest-contentful-paint",
"max-potential-fid",
"speed-index",
"total-blocking-time"
],
"budgetReadBy": [
".github/workflows/oshun-web-lighthouse.yml"
],
"evaluationScopes": [
"grounded_answers",
"citation_integrity",
"assistant_quality",
"assistant_safety",
"search_quality",
"recommendation_quality",
"persona_quality",
"voice_quality",
"avatar_quality",
"metis_pedagogy",
"generation_quality",
"generated_artifact_quality"
],
"evaluationStages": [
"select_dataset",
"run_inference",
"apply_metrics",
"compare_to_baseline",
"check_thresholds",
"publish_summary",
"attach_to_release_readiness"
],
"gateStatuses": [
"pass",
"fail",
"needs_data"
],
"missingMetricIsNeedsData": true,
"evaluateScopeRunUsedIn": [
"libs/oshun/analytics/src/__tests__/evaluation-manifest.test.ts",
"libs/oshun/analytics/src/evaluation-manifest.ts"
],
"pkceMethods": [
"S256"
],
"oauthCoreUsedIn": [
"libs/oshun/platform-foundations/src/platform-foundations.test.ts",
"libs/oshun/platform-foundations/src/public-api/oauth.ts"
],
"pageSaysNotADeployedServer": true,
"dependenciesStale": {
"saysRedis": true,
"saysS3InProd": true,
"namesComposePath": true,
"thatPathExists": false,
"mentionsValkey": false,
"lastChanged": "2026-07-16"
},
"drillsOnThePage": [
"Rollback drills",
"Data-residency drills",
"DSAR exercises",
"Tenant-isolation tests"
],
"drillsMustBeExecuted": true,
"manifestsDoNotProveDrillsRan": true,
"pageCounts": {
"scopes": true,
"stages": true,
"journeys": true,
"taraMinimum": true,
"sixDomains": true
}
}
Test runs on the working tree#
libs/oshun/i18n: npx vitest run (working tree, 2026-10-07)
RUN v4.1.7 /root/workspace/oshun-strong-verify-codex/libs/oshun/i18n
Test Files 1 passed (1)
Tests 17 passed (17)
Duration 286ms (transform 120ms, setup 0ms, import 135ms, tests 31ms, environment 0ms)
exit 0
libs/oshun/analytics: npx vitest run src/__tests__/evaluation-manifest.test.ts (working tree, 2026-10-07)
RUN v4.1.7 /root/workspace/oshun-strong-verify-codex/libs/oshun/analytics
Test Files 1 passed (1)
Tests 16 passed (16)
Duration 240ms (transform 104ms, setup 0ms, import 119ms, tests 10ms, environment 0ms)
exit 0
libs/oshun/platform-foundations: npx vitest run src/platform-foundations.test.ts (working tree, 2026-10-07)
RUN v4.1.7 /root/workspace/oshun-strong-verify-codex/libs/oshun/platform-foundations
Test Files 1 passed (1)
Tests 63 passed (63)
Duration 377ms (transform 202ms, setup 0ms, import 237ms, tests 15ms, environment 0ms)
exit 0
Valkey, Qdrant and MinIO in the Hetzner compose file#
git show b6f1ccf1c111:infra/hetzner/docker-compose.yml, lines 126 to 148:
126 # Valkey (BSD-3), not Redis 7.4 (RSALv2 or SSPLv1) — tracker T.01.08. The
127 # service keeps the name `redis` so every REDIS_URL below still resolves.
128 # `valkeydata` is a NEW volume: Valkey cannot read Redis 7.4's RDB v12, and
129 # the owner chose to flush at cutover (2026-09-15). What is lost is the
130 # BullMQ queue's in-flight jobs and the day's cost-ledger counters; drain the
131 # queue before deploying this. `redisdata` stays for rollback.
132 redis:
133 image: valkey/valkey:8.1.10-alpine@sha256:d2e18f3410b6f616de1417f570fa55261af2898b9c5b2cfb6781ce2373ea43d1
134 restart: unless-stopped
135 # volatile-lru: only keys with a TTL (idempotency, rate-limit, session
136 # state) are evicted under pressure; untagged data errors loudly instead
137 # of silently vanishing. AOF for restart durability.
138 command:
139 - valkey-server
140 - --appendonly
141 - "yes"
142 - --maxmemory
143 - ${REDIS_MAXMEMORY:-160mb}
144 - --maxmemory-policy
145 - volatile-lru
146 mem_limit: 256m
147 volumes:
148 - valkeydata:/data
git show b6f1ccf1c111:infra/hetzner/docker-compose.yml, lines 157 to 163:
157
158 # Derived personalization search index. PostgreSQL is authoritative: the BFF
159 # creates/verifies this collection and deterministically reconciles every
160 # point before it admits HTTP traffic. The volume accelerates normal restarts;
161 # clean-host recovery deliberately works from an empty Qdrant volume.
162 qdrant:
163 image: qdrant/qdrant:v1.19.1
git show b6f1ccf1c111:infra/hetzner/docker-compose.yml, lines 183 to 189:
183 minio:
184 # MinIO pinned by digest, from quay.io: the Docker Hub repository stopped
185 # answering anonymous pulls (a fresh host got "authorization failed" on
186 # 2026-09-15), and `latest` was a floating pin (tracker T.01.11). AGPL-3.0:
187 # run unmodified as a separate process, so §13 adds no source duty here; the
188 # licence text and CREDITS ship in the image.
189 image: quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
git show b6f1ccf1c111:infra/hetzner/docker-compose.yml, lines 216 to 217:
216 minio-init:
217 image: minio/mc:latest
The BFF waits for the four stores:
git show b6f1ccf1c111:infra/hetzner/docker-compose.yml, lines 322 to 333:
322 bff:
323 image: ${OSHUN_IMAGE_PREFIX:-ghcr.io/greychimp/oshun}/bff:${IMAGE_TAG:-staging}
324 restart: unless-stopped
325 depends_on:
326 postgres:
327 condition: service_healthy
328 redis:
329 condition: service_healthy
330 qdrant:
331 condition: service_healthy
332 minio:
333 condition: service_healthy
The file's header still describes the data tier as "redis 7":
git show b6f1ccf1c111:infra/hetzner/docker-compose.yml, lines 10 to 11:
10 # data tier postgres (pgvector/pg16, = RDS), redis 7 (= ElastiCache),
11 # minio (= S3; public S3 API via the edge at s3.<domain>)
The same images in the dev compose file#
git show b6f1ccf1c111:docker/docker-compose.dev.yml, lines 70 to 78:
70 # Valkey, not Redis: redis:7-alpine resolves to Redis 7.4, which is RSALv2 or
71 # SSPLv1 (tracker T.01.08). Valkey is BSD-3. The service keeps the name
72 # `redis` so every REDIS_URL still resolves. The data volume is new on
73 # purpose: Valkey cannot read Redis 7.4's RDB v12 files, and the owner chose
74 # to flush at cutover (2026-09-15) rather than replicate — dev state here is
75 # queues and daily ledgers. The old `redis_dev_data` volume is left in place
76 # for rollback.
77 redis:
78 image: valkey/valkey:8.1.10-alpine@sha256:d2e18f3410b6f616de1417f570fa55261af2898b9c5b2cfb6781ce2373ea43d1
git show b6f1ccf1c111:docker/docker-compose.dev.yml, lines 96 to 102:
96 minio:
97 # MinIO pinned by digest, from quay.io: the Docker Hub repository stopped
98 # answering anonymous pulls (a fresh host got "authorization failed" on
99 # 2026-09-15), and `latest` was a floating pin (tracker T.01.11). AGPL-3.0:
100 # run unmodified as a separate process, so §13 adds no source duty here; the
101 # licence text and CREDITS ship in the image.
102 image: quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
git show b6f1ccf1c111:docker/docker-compose.dev.yml, lines 137 to 138:
137 minio-init:
138 image: minio/mc:latest
What the deploy script does, in order#
git show b6f1ccf1c111:infra/hetzner/scripts/deploy.sh, lines 6 to 13:
6 # Usage:
7 # deploy.sh <staging|production> <image-tag> [service ...]
8 #
9 # <image-tag> normally the mutable per-env tag (staging|production) that CI
10 # just moved; pass an immutable sha-XXXXXXX tag plus an explicit
11 # service list to roll back a single service.
12 # [service ...] narrows pull/rollout to those services (CI passes the
13 # affected set). No list = full deploy of every service.
git show b6f1ccf1c111:infra/hetzner/scripts/deploy.sh, lines 26 to 37:
26 # 2. Sync edge config (Caddyfile + sites + compose) bundle → /opt/oshun/edge,
27 # bring the edge up, graceful `caddy reload`
28 # 3. Pull the target images
29 # 4. Bring up the data tier and wait for it to be healthy
30 # 5. Apply the ordered Oshun Prisma migration history via a one-off container
31 # on the stack network; untracked or failed migrations stop the rollout
32 # 6. Roll the application services
33 # 7. Wait for every target container to be healthy; on failure, dump the
34 # failing services' logs and exit non-zero (CI fails)
35 # 8. Public smoke through the edge (api/metis-api/psyche-api /health + apex)
36 # 9. Prune dangling images
37 # =============================================================================
The deploy workflow's flow and its production approval#
git show b6f1ccf1c111:.github/workflows/deploy-hetzner.yml, lines 10 to 12:
10 #
11 # Flow: plan (resolve env + affected services) → build (one GHCR image per
12 # service) → [production approval] → deploy (rsync bundle + run
git show b6f1ccf1c111:.github/workflows/deploy-hetzner.yml, lines 324 to 334:
324 approve-production:
325 name: Approve production
326 runs-on: ubuntu-latest
327 timeout-minutes: 60
328 needs: [plan, build]
329 if: needs.plan.outputs.environment == 'production'
330 environment:
331 name: production-approval
332 steps:
333 - name: Record approval
334 run: echo "Production deploy of image ${{ needs.plan.outputs.image_tag }} approved by ${{ github.actor }}" >> "$GITHUB_STEP_SUMMARY"
Launch locales and critical message prefixes#
git show b6f1ccf1c111:libs/oshun/i18n/src/index.ts, lines 26 to 35:
26 export const OSHUN_LAUNCH_LOCALES = [
27 'en-US',
28 'es-US',
29 'fr-FR',
30 'de-DE',
31 'ar',
32 'he',
33 'ja-JP',
34 'pt-BR',
35 ] as const;
git show b6f1ccf1c111:libs/oshun/i18n/src/index.ts, lines 99 to 99:
99 export const RTL_LOCALES: ReadonlySet<OshunLaunchLocale> = new Set(['ar', 'he']);
git show b6f1ccf1c111:libs/oshun/i18n/src/index.ts, lines 536 to 541:
536 const CRITICAL_MESSAGE_PREFIXES = Object.freeze([
537 'assistant.disclosure.',
538 'evidence.',
539 'auth.minimum_version_required',
540 'privacy.',
541 ]);
The evaluation result type#
git show b6f1ccf1c111:libs/oshun/analytics/src/evaluation-manifest.ts, lines 542 to 542:
542 export type EvaluationGateStatus = 'pass' | 'fail' | 'needs_data';
What the dependency registry still says#
git show b6f1ccf1c111:V1/DEPENDENCIES.md, lines 185 to 186:
185 `deploy/v1/hetzner/docker-compose.yml` and released by
186 `.github/workflows/deploy-hetzner.yml`. History: Vercel → Amplify → Fargate
git show b6f1ccf1c111:V1/DEPENDENCIES.md, lines 229 to 230:
229 | Redis | dev container | Event bus (`@oshun/event-bus`: Redis pub/sub with TTL-keyed replay — not native Streams), job queue (`@oshun/queue`), tenant-scoped cache, rate limiting, sessions. |
230 | MinIO | dev container; S3 in prod | Media (generated artifacts, Living Scenes renders, source ingestion, exports). Bucket per domain. |
Drills: required, and not claimed as run#
git show b6f1ccf1c111:V1/architecture/content-localization-and-launch.md, lines 171 to 187:
171 ### Operational drills
172
173 Each of the following must be _executed and signed off_ before general
174 availability — not merely planned:
175
176 - **Rollback drills.** Every surface that generates customer-visible artifacts —
177 Living Scenes workflow classes, generation tiers, the assistant, search — must
178 demonstrate an executable rollback to a prior version, with in-flight sessions
179 completing on the current version.
180 - **Data-residency drills.** Exercise primary-to-failover routing for each
181 launch region; confirm logs stay segregated by region and caches respect
182 residency boundaries.
183 - **DSAR exercises.** Run the full DSAR / deletion sequence end to end against a
184 seeded user, verifying tombstone propagation across every domain and audit
185 completeness before the receipt is issued. (The
186 `privacy.dsar_received`/`privacy.consent_revoked` strings above are the
187 customer-facing edge of this flow.)
The dev server's compose status, 7 October 2026#
Run in the checkout at e8a056602f5f862aeaf104c97edca6d1a96e1d85 on the Linux dev server. The
stopped dev containers are still redis:7-alpine and minio/minio:latest,
though the compose file declares Valkey 8.1.10 and a MinIO server pinned by
digest; and an anonymous pull of that pinned image is refused. No Docker
credentials are configured on the machine, and an anonymous token from quay.io
got the same refusal for the manifest and for the tag list. Nothing was started,
recreated or removed: an attempt to start the two services stopped at the pull,
before any container changed.
$ date -u +%FT%TZ
2026-10-07T22:49:09Z
$ docker compose -f docker/docker-compose.dev.yml ps -a --format "table {{.Service}}\t{{.Image}}\t{{.Status}}"
SERVICE IMAGE STATUS
mailpit axllent/mailpit:latest Exited (0) 2 months ago
minio minio/minio:latest Exited (0) 33 hours ago
pgbouncer edoburu/pgbouncer:latest Exited (0) 2 months ago
postgres pgvector/pgvector:pg16 Exited (127) 2 weeks ago
qdrant qdrant/qdrant:latest Exited (143) 2 weeks ago
redis redis:7-alpine Exited (0) 32 hours ago
$ grep -n "image: valkey\|image: quay.io/minio" docker/docker-compose.dev.yml | cut -c1-76
78: image: valkey/valkey:8.1.10-alpine@sha256:d2e18f3410b6f616de1417f570f
102: image: quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79
$ docker pull quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
Error response from daemon: unknown: failed to resolve reference "quay.io/minio/minio@sha256:14cea493d9a34af32
f524e538b8346cf79f3321eff8e708c1e2960462bd8936e": unexpected status from HEAD request to https://quay.io/v2/mi
nio/minio/manifests/sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e: 401 UNAUTHORIZED
What was not kept#
Nothing was deployed, so there is no log of a deployment. No drill record was looked for. The draft render's measurements and the axe results are in the record directory named above, not in this document.