--- title: Metis Disaster Recovery owner: unassigned — see "Who owns this" below version: 2.0.0 last_reviewed: 2026-09-18 next_review: 2026-12-17 applies_to: the Metis courseware service, its Postgres database and its source object bucket --- # Metis Disaster Recovery Written for M10.18.e from the shipped code paths, with every command below executed once against the compose Postgres while writing it. The measured numbers are from that run and are named as such; where something could not be executed, this document says so rather than describing what it would have done. **Revised 2026-09-18 (M10.17).** Three of the gaps version 1.0.0 recorded have moved, and one of its statements was wrong: - backups are now **scheduled and recorded** — `metis.tasks.durable.back_up_estate` runs every `backup_interval_seconds` and writes a row per attempt, so the recovery point is readable rather than folklore; - the object store half is now **exercised**, and it carries a constraint this document did not have: the objects must come back under the bucket name the receipts already name (see "Restore the object store"); - version 1.0.0 named the wrong buckets for blob recovery. The durable source bytes a revision's receipt points at live in **`metis-sources`** (`s3_bucket_sources`, M10.4) — not in `metis-courses`, `metis-assets`, `metis-exports` or `metis-uploads`, which hold derivatives, republishable exports, or objects a daily cleanup task deletes; - the whole procedure below is now executable as one test: `apps/metis/service/tests/test_estate_restore_drill_real_stack.py`. Prefer running it to reading this when what you need is confidence that a restore works; read this when you are performing one. ## What this covers, and what it does not Covered: taking a backup of the Metis database and its source bucket, restoring both into isolated copies, verifying each by hashing, and reading the recovery point off the estate itself. Not covered, because the capability does not exist yet: - **Point-in-time recovery.** `archive_mode` is `off` on the compose Postgres and `wal_level` is `replica`. A restore can only reach the state of the last dump. There is no procedure here for recovering to a chosen moment because there is no mechanism for it. - **Encrypted backups.** The dumps land in object storage unencrypted, and where a key would live is an operator decision nobody has made. Treat the backup bucket as holding everything the database holds. - **A stated recovery objective.** The schedule bounds the recovery point at `backup_interval_seconds` (default 86,400 — one day), but that is a default and not a commitment, and extrapolating the measured recovery _time_ needs a production volume nobody has recorded (M10.14.b). ## Who owns this Nobody, and that is a gap rather than an omission in this document. The estate has no accessibility or performance owner registry either (M10.13.f, M10.14.e), and this runbook cannot invent an on-call rota. Assign an owner before relying on it. ## Find the last successful backup first Before anything else, learn what you are restoring to. The estate records every attempt, and a failed run is in the table beside the successes so a gap in the schedule is visible: ```bash psql $H -d metis -c "SELECT finished_at, state, database_name, dump_bucket, dump_key, dump_bytes, tables, objects_copied FROM estate_backup_run ORDER BY finished_at DESC LIMIT 10" ``` The newest `succeeded` row is the recovery point. Its `dump_key` is the object to fetch from `dump_bucket`, and `dump_sha256` is what the bytes must hash to — check that before restoring anything, not after. If the newest success is old, or there are none, the schedule is not running: check that a beat worker is up and look for `estate_backup_unavailable` or `estate_backup_failed` in the logs. `unavailable` means a precondition — no `pg_dump` on the worker, or no backup bucket — and writes no row, which is why the log matters when the table is silent. ## Prerequisites - `pg_dump`, `pg_restore`, `createdb`, `dropdb` on the path. - Credentials for the compose Postgres. In development that is `oshun:oshun_dev` at `127.0.0.1:5432`; never put the password in a committed script or in command text — export `PGPASSWORD` from a throwaway file and delete it afterwards. - Enough disk for the dump. The development database was 1.1 MB compressed; production volume is unrecorded, which M10.14.b is blocked on. ## Take a backup ```bash export PGPASSWORD=… # from the operator's secret store, not here H="-h 127.0.0.1 -U oshun" pg_dump $H -Fc -f /tmp/metis-$(date +%Y%m%dT%H%M%SZ).dump metis ``` `-Fc` is the custom format, which `pg_restore` can restore selectively and in parallel. Measured: **204 ms** for the 1.1 MB development database. ## Restore into an isolated database Never restore over the database you are recovering from. Create a new one and compare before switching anything. ```bash dropdb $H --if-exists metis_restore_drill createdb $H metis_restore_drill pg_restore $H -d metis_restore_drill --no-owner --no-privileges ``` Measured: **1,981 ms**, total **2.2 s** end to end. That is a floor for a development-sized database and says nothing about production. `--no-owner --no-privileges` drops grants. On the development database that loses nothing — it carries zero role grants on `public` beyond the owner and `PUBLIC`, checked with: ```bash psql $H -d metis -tAc "SELECT count(*) FROM information_schema.role_table_grants WHERE table_schema='public' AND grantee NOT IN ('oshun','PUBLIC')" ``` On a deployment that grants to real roles, drop those two flags and restore as a superuser, then re-run the verification below. ## Verify the restore by hashing, never by counting A row count says two tables agree whenever they hold the same number of different rows. Hash the content instead: ```bash for DB in metis metis_restore_drill; do : > "/tmp/hash-$DB.txt" for T in $(psql $H -d "$DB" -tAc \ "SELECT tablename FROM pg_tables WHERE schemaname='public' ORDER BY 1"); do HASH=$(psql $H -d "$DB" -tAc \ "SELECT COALESCE(md5(string_agg(x::text, E'\n' ORDER BY x::text)),'EMPTY') FROM \"$T\" x") echo "$T=$HASH" >> "/tmp/hash-$DB.txt" done done diff /tmp/hash-metis.txt /tmp/hash-metis_restore_drill.txt && echo IDENTICAL ``` Measured: **163 tables hashed on both sides, all 163 digests identical**, with 18 tables non-empty. Check that last number — a comparison in which every table is `EMPTY` passes and proves nothing. A second, independent measurement on 2026-09-18, from the drill test on a freshly provisioned stack carrying one source-to-course journey: **172 tables, all digests identical, 7 non-empty**, dump 938,995 bytes. Dump **0.181 s**, restore **1.790 s**. One table legitimately differs when you restore a dump this estate's own scheduled backup produced: `estate_backup_run`. The run row is written after the dump completes, so a dump cannot contain its own record. Every other table must match. Tenant integrity needs no separate step: `tenant_id` is part of every row that was hashed. ## Restore the object store **The objects must come back under the bucket name they were stored under.** This is not a preference. A revision records each blob as a `BlobReceipt` that carries its bucket, and `BlobStore.verify` reads the bucket out of the receipt rather than out of configuration — so objects restored into a differently-named bucket are objects nothing can verify, and the receipts cannot be edited to suit: they live in `store_revisions`, which migration 047 makes immutable outside an approved retention or legal workflow. Measured 2026-09-18: with the objects mirrored to a second bucket and the original emptied, the reconciler reported **every blob broken** — and reported them broken again when pointed at the bucket that held them. Restoring into the original name made the estate reconcile. The scheduled backup keeps the source objects under the `objects/` prefix of the backup bucket, keyed exactly as they are in `metis-sources`. Restoring them is `estate_backup.restore_sources`, and it is a function rather than a shell line for two reasons. Neither `mc` nor the `aws` CLI is installed on the hosts this estate runs on (checked 2026-09-18 — `which aws mc` finds neither); either could be installed, but the function is the one the service's own tests exercise, and a restore procedure that is only ever run by hand is a procedure nobody has verified. ```bash cd apps/metis/service PYTHONPATH=src .venv/bin/python -c " from metis.config import get_settings from metis.services.durable_runtime import s3_client from metis.services.estate_backup import restore_sources settings = get_settings() print(restore_sources(s3_client(settings), settings, settings.s3_bucket_backups)) " ``` It restores into `s3_bucket_sources` and nowhere else, refuses a bucket that is not there rather than creating one, and is idempotent — an object already present under its content-addressed key is the object the backup holds. It prints `(objects_restored, bytes_restored)`. Rehearsed 2026-09-18 against MinIO with one object in a backup bucket: the command above printed `(1, 27)`, and running it again printed `(0, 0)`. Then verify the bytes rather than the listing — every object's length and digest, on both sides. `tests/test_estate_restore_drill_real_stack.py` and `tests/test_estate_backup_real_stack.py` do exactly that, and running them is the faster way to check the procedure still holds. Measured: mirror **0.029 s**, restore **0.045 s** for 2 objects / 1,723 bytes. That is a floor at development volume and nothing more. ## Reconcile after the restore A restored database holds outbox events nobody delivered and projections nobody has compared. Drain and reconcile before switching traffic: ```bash # the worker task beat runs every 10 seconds; run it once against the restore METIS_DATABASE_URL= celery -A metis.tasks.celery_app call \ metis.tasks.durable.drain_outbox METIS_DATABASE_URL= celery -A metis.tasks.celery_app call \ metis.tasks.durable.reconcile_estate ``` `reconcile_estate` repairs nothing; it reports. A clean pass means the blobs, the search index, the outbox, the published exports, the grade projections and the aggregate store's own record legs all agree with the restored record. Any finding is for a person to act on before the estate is declared recovered. ## Clean up ```bash dropdb $H --if-exists metis_restore_drill rm -f /tmp/metis-*.dump /tmp/hash-*.txt ``` Leaving a restored copy on the server is how a stale database gets mistaken for the live one. ## What to do when the verification fails Do not switch traffic. A digest mismatch means the dump, the restore or the source changed between the two hashes — take a fresh dump with the service stopped and repeat. If the mismatch persists on a quiesced source, the dump is suspect and the correct action is to escalate rather than to accept the closest copy. ## Known gaps, as of the last review | Gap | Consequence | | ------------------------- | ----------------------------------------------------- | | No WAL archiving | No recovery to a chosen point; only to the last dump | | Dumps are not encrypted | The backup bucket holds everything the database holds | | Interval is a default | 86,400 s is configured, not committed to | | No owner | Nobody is accountable for running any of this | | Production volume unknown | The measured RTO cannot be extrapolated | Closed since version 1.0.0: the schedule (`back-up-estate`, with every attempt recorded in `estate_backup_run`) and the object-store half (mirrored by the backup, restored and verified by the drill test).