Implementation task: platform-ops:fleet-automatic-recovery-policy-2026-10-04. Requested by Elikem on 2026-10-04. This runbook supplements the worker and reviewer operation guide; it does not grant new authority or change an automation switch.
Implementation and deployment status#
Current verified selection as of 2026-10-04 15:28 UTC: product, platform and delegate select pushed and tested 12b635c36d5a34d3d2193313eab5cf531cdc5ae8; reviewer selects a28e3d9429a51e660309ee3ae9c54eba3bada7fd. The API selects a866be9a722f946d0617f46a17c8fd665c16d479. Installed controller files match 5b1328c7275f480696fa194300f69cefc96894a7 and its current normal invocation began at 14:43. Active jobs finish their existing source cycle. Current receipts and limitations are below. Earlier unsupported commissioning statements remain historical revisions, not acceptance evidence. Independent review e15e76a9-4072-4279-a43f-3c83c1dbfb3d accepted the dated runner/rollback receipts but requested attributable resource authority and controller/CI records. Revision 15 supplied the controller/CI records. Review 9370990f-d86a-4f88-8509-be00d48639a9 of revision 15 accepted them and asked for two things, which this revision supplies: the owner's attributable resource-policy decision (Elikem's approval of operation guide revision 51; see Platform resource allocation and owner approval) and an authenticated completion receipt for HOME (see Recovered jobs and preserved work).
Recovery contract#
Each job retains its subject, authenticated owner, fencing token or immutable review/decision pin, last completed progress, exact pending request payload and retry identity, checkpoint location, cause, attempt count, elapsed budget and next action. An interrupted mutation is replayed with the same identity and payload before another effect is attempted. Atomic replacements fsync the file and its directory. Invalid records are preserved for repair instead of treated as fresh jobs.
A heartbeat says the process can communicate. Completed tool or stage progress says it is making progress. These are separate signals. Successful recovery still requires the task's normal checks and independent review. A repaired dependency permits another verification attempt; it does not make a test pass.
Bounded policy#
Budgets persist across process restarts and changed error wording. Each failure class and verified evidence version has its own ledger. Transport failures allow at most six failed attempts within six hours, starting at one minute. Provider capacity allows four in 24 hours, starting at 15 minutes. Provider account exhaustion uses its existing quota probe; model overload does not claim account exhaustion or change the selected model. Typed top-level Codex server-overload errors, or the exact observed no-code provider capacity message, are normalized before applying this budget. Agent/tool prose and conflicting authority codes do not qualify. Resource failures allow three in six hours, starting at five minutes, and the live resource guard must pass before work starts. Interrupted execution allows three in six hours, starting at one minute. Invalid structured output allows two in one hour, starting at one minute. Backoff doubles with jitter and caps at one hour.
Deterministic defects and unknown failures wait for a verified repair after their first failure. Authentication, scope, explicit cancellation and corrupt checkpoint failures require accountable handoff. A tested runner revision or a completed typed blocker can wake an eligible job, with no more than three changed-evidence wakeups. Passage of time, a new error message and a service restart cannot grant a new budget.
Product and platform workers#
New version-two checkpoints retain the task definition and dependency identities before the claim. Recovery verifies the same identity and definition, dependencies, complete question journal, pause/cancel state and current lease. The server independently enforces automation switches, family routing and one active claim per identity. General recovery reconciles and releases only its own stopped lease, then obtains a valid fresh fence. An active different owner, superseding work or changed task definition requires reconciliation. A narrow automatic no-work handoff is accepted only from a complete authenticated journal: original release, one matching foreign claim/release, exactly two version changes, no intervening work or edits, unchanged task definition, and a ready unclaimed task. Recovery then claims a fresh fence and verifies the resulting version and owner. A durable pending-wakeup marker survives a crash without resetting its budget. Existing quota recovery retains its proven replay and expiry rules.
A worker can record typed blockers in the task's ignored .cache/eve-worker/<task>-<fence>/recovery-blockers.json as an array of {kind: task, key: family:id} objects, with string values. The runner reads versions and completion states from Eve. All recorded blockers must be shipped or verified before their changed evidence permits a retry. When no typed blocker exists, a stable platform-ops repair incident supplies the handoff and wakeup link.
Stopped checkouts, index entries, untracked files and unpublished commits remain at their original paths. A retained Git ref protects the last checkpointed commit. Separate linked worktrees let unrelated eligible tasks proceed. The scheduler allows at most twelve retained slots and needs ten GiB free before allocating another. It never discards work to gain capacity. Pending board writes must match the saved endpoint, identity, task and fence; a pending completion also needs its exact commit on origin/main.
Legacy or unattributed checkouts are preserved and routed for reconciliation. Ordinary code conflicts may be repaired by the worker within the same authorized task after fencing; recovery does not reset or clean the checkout. Explicit pauses and cancellations are not automatically undone.
Reviewer and delegate#
Claims, completions, decisions and release acknowledgements use durable receipts. A restarted runner replays the pending operation before selecting another subject. Expired or refused outcomes remain archived. Reviewer and delegate recovery pins the exact immutable subject; it never substitutes a newer document or authors another decision to resolve a lost acknowledgement.
A blocked job's ledger allows another eligible queued job to proceed through an explicit-ID claim. Malformed output consumes a bounded budget; repeated missing evidence remains visible. The review queue's trusted-evidence retry changes queued_at and may wake the same pinned review. A tested harness revision may also permit bounded re-verification. Invalid or incomplete reports never become approval. Reviewers still do not implement fixes, self-review or publish.
CI/CD#
The controller retains existing deployment locks, exact commit and frozen-content pins, checkpoint validation, release receipt replay and rollback rules. A failed fast tier or named verification command has two attempts; a general release failure has three, bounded by 24 hours. Structured transport failures (connection errors or SSH/SCP exit 255) have six attempts with a five-minute cooldown, also bounded by 24 hours. Error prose alone cannot classify an infrastructure failure. Per-stage budgets survive alternating stage failures. A new main commit has its own verification budget. A read-only probe of the frozen Dev selection may wake a content-related failure at most three times; the probe never freezes, approves, rebinds or promotes content. Host memory/disk deferrals start no deployment and remain governed by resource preflight.
Existing release failure records and the blocked-release watcher route deterministic failures and urgent release incidents. A failed browser or content check remains blocking until repaired and verified. Recovery grants no UAT promotion, publication or approval authority.
Independent supervision and escalation#
Each agent host runs an independent 60-second systemd recovery timer using its existing scoped identity. It observes meaningful progress and the systemd invocation ID. A worker stalled for 90 minutes, or reviewer/delegate for 60 minutes, receives a bounded stop only when the checkpoint PID and invocation match the active process and the host already has authorized full access. An old uninstrumented process is reported rather than killed. At most two interventions are allowed until a tested runner revision or verified healthy completed cycle changes the evidence.
The platform host's independent watch reads fleet heartbeat and attention state. Stale heartbeats over ten minutes or persistent recovery attention produce a deduplicated platform-ops incident through a durable outbox. It records healthy observations but does not approve or close its own repair task. API outages preserve pending reports. Unknown failures retain their evidence and assigned repair owner; owner-only decisions remain Elikem's.
Rollout and rollback#
Install only a pushed and tested source revision in a protected root-owned runner-releases/<full SHA> directory. Record each prior effective source, configuration and systemd override before selecting the new revision. Validate units and watchdog timers. Active jobs finish their existing cycle; do not restart them for rollout. Observe the next natural cycle, fresh heartbeats, preserved checkpoints and independent watch results on each host. Keep the existing scoped credentials and automation switches.
Rollback restores the recorded runner configuration and effective source override, validates units, reloads systemd and takes effect at the next natural cycle. For a source-selection-only rollback, keep the existing watchdog timer. A full rollback to a pre-watch installation may disable only the newly added timer, using its recorded prior state; that timer-disable path was not exercised by the dated selection-only receipt below. Preserve version-two records and linked worktrees: an older runner may halt on them and requires explicit reconciliation. Never delete a checkpoint to make rollback appear healthy. The CI controller's protected prior source remains available for the same rollback; keep its failure receipts.
Verification and limits#
Failure injection covers lost acknowledgements, replay identity, expired and superseded fences, changed definitions and dependencies, open questions, switches, cancellation, retained dirty work and unpublished commits, resource guards, quota, malformed output, corrupt records, elapsed and attempt budgets, and watchdog/API outages. The overnight regression is integration failure, lease expiry, blocker repair and automatic resume with a fresh fence, without operator edits to the checkpoint. Verification evidence and rollout observations belong in the implementation task journal.
Universal repair of arbitrary unknown failures is not promised. Credential changes, spending, owner-only decisions, unsafe ownership changes and unknown damaged state require their authorized repair route. A total host or network loss remains visible to the independent fleet monitor but cannot be repaired by software executing on that unavailable host.
Platform resource allocation and owner approval#
The measured platform MemoryMax is 11,524,898,816 bytes and next-cycle Node heap is 8192 MiB. The installed half-workload allocation, capped at 12 GiB, comes from implementation 4cb7e11f750c60a45e63b993055e9209d3fabf21. These host readings establish configuration; the approval below establishes authority.
Elikem approved operation guide aaf9cfbb-447f-4254-a519-da29d3d78f07, revision 51 on 2026-10-05 at 11:36:12 UTC. Its content hash is 6b93f4ecab3f0e75ac3b64e45fd3365158b576c99e23b9ca4bc98fd4a2c5d764 and its reviewer of record is his user account, 666d8bfd-4497-4fd0-87c6-a391c5aa8440. That revision's "Current platform memory instruction" section says Codex receives at most half the workload memory budget, capped at 12 GiB, on the shared platform host. Its platform-worker section records the same 11,524,898,816-byte cgroup budget and 8192 MiB heap measured here, with a dated host receipt from 2026-10-05T11:02:34Z. This runbook pins that revision as a reference. His assistant carried out the approval in Elikem's own Workspace session after his explicit instruction, under his 2026-09-29 decision that his assistant may carry out his approvals; the decision and the record are his. The approval records the policy. It changes no host setting, and this runbook still cannot itself authorize an allocation change.
This record supersedes revision 15's account, which relied on the owner's conversational "Approve revision 45" and noted that the Chrome access attempts were refused. The submitted operation-guide revision 45 (SHA-256 267a55c8bbb9189ae73ad52d891177e51f1b0d776c9ec902ef44e8e8ddf1474f) and the full-budget October 3 revision 43 remain historical evidence of how the conflict arose. Repository bootstrap commit 81808f47f2b9bab1b73aab63ec86b605449b149f aligns AGENTS.md and both platform roles with the same instruction, while preserving CPU/sudo authority and the 2 GiB OS reserve. Its 16 resource-policy tests, formatting, strict staged hooks and full-SHA preflight passed.
Dated operational evidence#
Runner tests, installers, host readbacks and rollback receipt: 73,848 bytes, SHA-256 3a128ed95927e993d4e9c8b18efeb15140d36c94475046ede05411f7ee070ec0. The attachment includes the actual health/rollback helper sources and outputs, without credentials. Installed runner archive SHA-256 is 124617fc6e157810938da1c876df9098f02d9d84baf21a26db2f1406d1abe9fb.
At14:33:48 UTC all four recovery-watch timers were enabled and active with Result=success. Effective service memory ceilings were: product14,220,787,712; platform11,524,898,816; reviewer23,050,846,208; delegate23,050,846,208 bytes. Product, platform and reviewer had recent completed command progress. The delegate's prior decision cycle was finished and normal idle cycles continued. Service type=oneshot reports activating while its runner executes; this is not itself a startup fault. These observations prove current supervision and progress, not completed product verification.
At14:26:30–14:26:34 UTC an operator held the delegate cycle lock, selected retained d9bdb395, and restored12b635c. Configuration SHA-256 returned exactly to72d1bbda8b82855ce88fc8b1acf75e13fb338d2ff81d411060c2141103dea28f; source-override hash returned exactly toc3d5b4ed4d628157b81a21418cc0d1ecae0c502f3e5496e71781704a34df9090. MemoryMax and watch timer stayed unchanged. No model job was started or interrupted: this verifies configuration selection and restoration, not model execution on the rollback source. A scheduled invocation collided with the operator lock and failed; after its release the normal timer completed successful recovered/idle cycles at14:31 and14:32, with journal receipts attached. The earlier undocumented claim of a full delegate rollback is withdrawn as acceptance evidence.
Recovered jobs and preserved work#
Handoff repair b4230a318ede589e7bdf4919d03a7a7b32f35b26 plus wake-budget validation ee71c236524bb7b75f347dab37c4cf632f9d48ef passed62 targeted Node tests and6 real PostgreSQL/API tests, all zero failures/skips. The help task automatically resumed under fresh fence3 at14:11:55 and shipped78b14c65f09ab4c4054d8485a3eebb9e4d2da388 at14:22 after26 CLI tests. Its separately attributed platform acceptance/regression is media3ccec6aa-bf8e-8e38-970c-98156b042225, pushed dcb19376c5be2bf1b6ece01ee34764151e688db4. HOME (eve-gap-analysis:home-src-human-video-review-obligations-2026-10-03) automatically claimed fresh fence 3 at 14:24:39.634 after all three actual prerequisites shipped. eve-codex-worker-01 then recorded done at 14:42:09.689 under fence 3 with commit 0f2f3a89165b3e77fd7c917224571fe6d78df555. Authenticated HOME lifecycle receipt, SHA-256 f274901ebe6df09d15e3f0e4ed007fd5cead3a7c3e1dcc0d8981a651bee73987, read on 2026-10-05 through the scoped claude-mac-worker identity, shows the task shipped at version 8. Its closure is by done, actor eve-codex-worker-01, at 14:42:09.689Z, commit 0f2f3a89165b3e77fd7c917224571fe6d78df555. The receipt includes the whole non-activity journal from creation: claims under fences 1, 2 and 3, releases, notes, done and session close. The commit is an ancestor of origin/main (checked 2026-10-05 against 7aa3364122218c6fe86f3336564332a945fb7e38). The closure's verified flag is false: Eve records the transition and commit, not independent verification of the task, which follows its own review. The earlier preserved-attempts attachment predates this transition, which is why it shows the task leased.
Final preserved-attempt dispositions, SHA-256 1e516c543b7d598ef79c1a335a6cb79d4da2b3ff64c626f6fdf37e4e83ff016a: c4922170 presentation-notes,6cad292e Demeter dependency and0d84bc9e platform reviewer-fix attempts were reconciled under the relevant cycle lock. Each original commit, branch, dedicated retained ref, full patch and checkpoint survives. The Demeter and reviewer implementation blobs/modes matched authenticated separately shipped receipts. The reviewer task's sole original self-authored scope edit was explicitly checked. The note-probe's unique22-line test was preserved verbatim in b512d5a8b135369f6577301e3a7d48871e9b6b5d and passed13 private PostgreSQL tests. Demeter's unique verification file was archived; its separate mobile assertion failures are not called fixed. Platform readback independently verified its final14:36:40 disposition after the initial SSH acknowledgement stalled. No foreign completion was attributed to the old worker fence.
Provider model-capacity repair12b635c36d5a34d3d2193313eab5cf531cdc5ae8 passed128 Linux runner/reviewer/recovery tests and31 delegate tests including real PostgreSQL/API scenarios, all zero failures/skips. Targeted lint, format, staged hooks and content preflight passed. The original provider event was recorded in session f5699479-6fd4-48f7-94ac-11d6654637b2 at13:39:45 and13:39:51. Its execution-isolation task is now owned by claude-board-worker fence4; the old Codex fence3 is invalid and is not resumed. Its13 dirty/untracked files remain preserved in slot efc14d52-6c8e-46b5-8798-01aebe4776d9 until the independently owned successor can be reconciled. Capacity classification is repaired; that product implementation/review is a separate unfinished gate.
API, graph and CI evidence#
Snapshot acceptance on original host204.168.220.77, SHA-256 cfbc39944c66230f368facb985a5fb218c7d7ac04397e364443340ba8b076999: two ordinary ./eve dump --stable calls using the operator's own identity and normal API limits exited0 after91.276s and61.167s. Reader framing, digest and counts passed. Output sizes239,141,063 and239,154,026 bytes; full hashes and root-only compressed retention receipts are attached. The12:58:16 API log explicitly records export-board503 snapshot_timeout; the exact internal phase of the historical October3 failure is not claimed. Snapshot deadlines/compression passed121 Linux PostgreSQL/API/client/export/reader/transfer tests. Earlier failed uncompressed and setup attempts remain evidence, not passes.
Preflight a866be9a722f946d0617f46a17c8fd665c16d479 now refuses nonexistent commits instead of falsely accepting them;12 Git preflight and2 pin-guard tests passed. Previous focused test receipts. Reviewer replay repair freed unrelated reviews while preserving refused approval requests and their exact pins; blocked prerequisite approvals remain subject to normal review. Live graph repair resolved all222 slide links, with13 clean checks and0 problems;13,464 historical/shape warnings remain. Publication and policy did not change.
The normal Dev release of 8c6314b10fbf65c7619e4a8d4ce5cc1f4f84d529 passed its API stage at 13:42:56: 822 passed, 0 failed, 7 recorded legacy skips. The dated journal summary is attached below. That release later failed its browser gate at 14:41: 132 passed, 1 failed, 1 skipped. It stopped before deployment and did not replace the API hotfix. The presentation case combined two viewport scenarios under one 30-second deadline. Commit a119952771a5ee0d0ac0c102ccc9c28900aaaed4 gives desktop and mobile independent cases, preserving all assertions, each 30-second deadline, the 5-second rendering limit and zero retries. Browser regression receipts, SHA-256 8b12f38419b974fb8a9a0d5f57a555a9849c18ac2f5f42ee528e16d299dde94c: six repeated cases passed in 13.8–21.9 seconds each, plus app typecheck, lint, formatting, staged hooks and preflight. This targeted proof does not replace the normal full gate.
The next normal release selected 26685967b60cf0bfd826a93e2d1d9c03a7bbfceb at 14:43. Fast Node and web typecheck passed; affected Nx tests exhausted their 1200-second budget and Nx typecheck/lint exhausted their 480-second budget. Both are recorded as incomplete advisory checks, not passes. At 15:25 the controller had prepared its immutable source checkout and continued normal verification. It predates the browser split and latest reviewer repair. No final application rollout is claimed. UAT promotion, content publication, owner-only decisions and independent approval remain outside automatic recovery authority.
Reviewer alert recovery and controller evidence#
Late verification, deployed controller hashes and dated CI API summary: 95,455 bytes, SHA-256 dd8a2336fde5289069a4ac181da9a61510b607ffafe3e5d227d2670191c78a78. At 15:18 the installed release.py hash was a52025c4bf7b9f107569e26e1103bced1569ae637f452555a29e69f2af5e5d37 and recovery_policy.py hash was 5ca4c3062dbe1bf99ac75e42b49adb09d1109729b9464882544c6ce258d3c182. Both exactly match their files in pushed 5b1328c7275f480696fa194300f69cefc96894a7. The systemd readback shows the normal controller invocation began at 14:43:21. These observations establish installed source and invocation, not completed rollout. The attachment includes the actual dated 13:42:56 API test summary from controller PID1874446.
Reviewer commits 670cf0f89e2f2bf970f0e09183c2364977aa222f and a28e3d9429a51e660309ee3ae9c54eba3bada7fd correct stale execution attention after a genuine needs-evidence report. Retirement requires exact immutable subject, own reviewer, later fence and time, no unresolved writes, and a durable archived failure receipt; technical report failures and ambiguous provenance retain their budgets. At most ten exact retained IDs are checked per cycle with a durable rotating cursor, independently of the missing-evidence backlog. The final Linux reviewer/recovery and real PostgreSQL review-loop suite passed 118 tests with zero failures/skips. A preceding fixture expectation failure was corrected before that final result; its run is not counted as a pass.
The normal 15:13 cycle automatically archived review 02a6c565-f647-4253-ace2-268a3c7b9946's obsolete transport failure, kept its valid substantive needs-evidence result, and processed b6600df1-b7f7-4870-9792-95ac6b7dbf22. The 15:28 authenticated API receipt shows reviewer heartbeat 15:27:54 with null attention. The archive hash is 99fdbdac2faee0904333db198b4712ade01c8ecde6467c39f546096ccac0ac30. No reviewer job was interrupted and no missing author evidence was waived.