# Worker recovery: retained implementation and verification evidence

Collected by codex-eve-migration on 2026-09-24. This is an operator-authored evidence record, not independent approval. It contains exact code excerpts, sanitized observed configuration and retained unit-test stdout. Tests using mocked clients/processes establish the named boundaries; they do not prove a particular live recovery succeeded.

Historically observed worker source commit: 52f74037f6ec233e542794f946f7197ef686e09e, as asserted by the retained 2026-09-24 configuration observation below. This is not a current installation attestation. On 2026-10-07, a fresh local test run used an unmodified Git archive of that exact commit. Its resources module, recovery module and installer bytes match the SHA-256 values below. This establishes applicability to the historically recorded source revision, conditional on the original installation observation; it does not establish current deployment or successful live recovery. The conflicting 47/46-test provenance is withdrawn as verified evidence and preserved explicitly below.

## tools/eve-reviewer/resources.mjs

SHA-256: ee0dcbe34384db9548c3a4e094d340288b70649a8f5eca5b415c6c914c945b98

```js
import { readFileSync, statfsSync } from 'node:fs';
import { join } from 'node:path';

const GiB = 1024 ** 3;

export function workerBudget(config) {
  const memoryGiB = config.memoryGiB ?? 4;
  if (!Number.isInteger(memoryGiB) || memoryGiB < 4 || memoryGiB > 12)
    throw new Error('Invalid worker memory budget');
  // Odd-sized service budgets add native/process headroom without increasing
  // the Node heap. Keep the established four-GiB profile and bounded heap sizes.
  const nodeHeapMiB =
    [2304, 4096, 6144, 8192, 10240].filter((heap) => heap <= memoryGiB * 1024 - 2048).at(-1) ??
    2304;
  return { memoryGiB, nodeHeapMiB };
}

export function workerResources(directory) {
  const available = /^MemAvailable:\s+(\d+)\s+kB$/mu.exec(readFileSync('/proc/meminfo', 'utf8'));
  if (!available) throw new Error('Cannot read available host memory');
  const fs = statfsSync(directory);
  const group = /^0::(\/[^\n]*)$/mu.exec(readFileSync('/proc/self/cgroup', 'utf8'));
  let currentBytes = null;
  if (group) {
    const path = join('/sys/fs/cgroup', group[1], 'memory.current');
    try {
      currentBytes = Number(readFileSync(path, 'utf8').trim());
    } catch {
      /* A host without cgroup v2 still has the host-memory guard. */
    }
  }
  return {
    availableBytes: Number(available[1]) * 1024,
    freeDiskBytes: fs.bavail * fs.bsize,
    currentBytes,
  };
}

export function resourceRefusal(sample, budget, { starting = false } = {}) {
  const minimum = (starting ? budget.memoryGiB + 2 : 2) * GiB;
  if (!Number.isFinite(sample.availableBytes) || sample.availableBytes < minimum)
    return starting
      ? 'Waiting for enough host memory to start the worker safely'
      : 'Host available memory fell below 2 GiB';
  if (!Number.isFinite(sample.freeDiskBytes) || sample.freeDiskBytes < 2 * GiB)
    return 'Worker filesystem has less than 2 GiB free';
  if (
    !starting &&
    sample.currentBytes !== null &&
    sample.currentBytes > (budget.memoryGiB - 0.5) * GiB
  )
    return 'Worker approached its service memory ceiling';
  return null;
}

```

## tools/eve-reviewer/worker-recovery.mjs

SHA-256: 2338ff8939e7db63000a65c32239c318665b2e925b641aaee3ff6c8ca134fe73

```js
import { existsSync, readdirSync } from 'node:fs';
import { join } from 'node:path';
import { questionsAndAnswers } from '../eve-task-board/lib/context.mjs';

/** Explicit operator recovery; normal timer polls never discard an interruption. */
export function resumeArguments(args) {
  if (!args.length) return null;
  if (
    args.length !== 4 ||
    args[0] !== '--resume' ||
    args[2] !== '--token' ||
    !args[1].includes(':') ||
    !/^[1-9][0-9]*$/u.test(args[3]) ||
    !Number.isSafeInteger(Number(args[3]))
  )
    throw new Error('Use --resume <task-key> --token <fencing-token>');
  return { key: args[1], token: Number(args[3]) };
}

export async function readWorkerResume({
  client,
  saved,
  resume,
  identity,
  repository,
  git,
  now = Date.now(),
}) {
  if (
    saved.agent !== identity.agent ||
    saved.key !== resume.key ||
    saved.token !== resume.token ||
    !['running', 'needs-attention'].includes(saved.state)
  )
    throw new Error('Recovery must match the saved worker identity, task and fencing token');
  const pending = join(repository, '.cache/eve-task-board/remote-requests');
  if (existsSync(pending) && readdirSync(pending).length)
    throw new Error('Resolve pending Eve requests before resuming preserved work');
  if (git(['ls-files', '--unmerged']))
    throw new Error('Resolve the preserved Git conflict before resuming');
  const current = await client.readTasks({ command: 'show', key: saved.key });
  const claim = current.claim;
  if (
    !['leased', 'in-progress'].includes(current.task?.status) ||
    claim?.agent !== identity.agent ||
    claim.fencing_token !== saved.token ||
    !Number.isFinite(Date.parse(claim.expires_at)) ||
    Date.parse(claim.expires_at) <= now + 60000
  )
    throw new Error('Recovery requires this worker’s live, unchanged lease with time remaining');
  const events = [];
  let cursor = null;
  for (let page = 0; page < 20; page++) {
    const history = await client.readTasks({
      command: 'history',
      key: saved.key,
      limit: 100,
      ...(cursor === null ? {} : { cursor }),
    });
    if (!Array.isArray(history.events))
      throw new Error('Cannot verify the interrupted task journal');
    events.push(...history.events);
    if (history.nextCursor === null || history.nextCursor === undefined) {
      const questions = questionsAndAnswers(events.sort((a, b) => Number(a.id) - Number(b.id)));
      if (questions.some((question) => question.answer === null))
        throw new Error('The interrupted task is waiting for an answer from Elikem');
      return {
        task: { ...current.task, key: saved.key },
        lease: { token: saved.token, expiresAt: claim.expires_at },
        questions,
      };
    }
    cursor = history.nextCursor;
  }
  throw new Error('The interrupted task journal exceeds the recovery read bound');
}

```

## Installer selections

infra/hetzner/eve/install-agent-runner.py, SHA-256 6b53912afe3d339f5a212a1753484034a3581a58f8516b863d3893e1f4f49079. Relevant exact lines:

```python
MemoryMax={str(worker_memory_gib)+'G' if role == 'worker' else '1536M'}
MemorySwapMax={'2G' if role == 'worker' else '1G'}
    parser.add_argument('--worker-memory-gib', type=int, choices=range(4, 13), default=8,
```

## Observed live configuration

Captured 2026-09-24T01:03:02.182257+00:00, read-only SSH/systemd inspection of 204.168.220.77. This is a point-in-time observation; the runner configuration is operator-controlled. No credentials are included.

```json
{
  "model": "gpt-6-sol",
  "reasoningEffort": "xhigh",
  "memoryGiB": 11,
  "repository": "/srv/eve-codex/worker/repository",
  "service": "ExecStart={ path=/usr/local/lib/eve-codex/bin/node ; argv[]=/usr/local/lib/eve-codex/bin/node --max-old-space-size=768 /srv/eve-codex/runner-releases/52f74037f6ec233e542794f946f7197ef686e09e/tools/eve-reviewer/worker.mjs ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }\nMemoryMax=11811160064\nMemorySwapMax=2147483648\nUser=eve-codex-worker\n"
}
```
The 11 GiB budget computes an 8192 MiB child heap. MemorySwapMax is 2 GiB. The recovery/resource modules above establish the startup and runtime thresholds.

## Retained verification log

Fresh verification, 2026-10-07: tested Git commit 52f74037f6ec233e542794f946f7197ef686e09e. Command: `node --test tools/eve-reviewer/delegate.spec.mjs tools/eve-reviewer/resources.spec.mjs tools/eve-reviewer/reviewer.spec.mjs tools/eve-reviewer/worker-recovery.spec.mjs`. Working directory: private unmodified archive of that commit's tools/eve-reviewer, tools/eve-task-board, tools/eve-content and tools/eve-git-integrate directories. Exit status: 0. Result: 29/29 passed, 0 failed, 0 skipped. Retained stdout/stderr SHA-256: `dc6679d212a6be2ebfaaf399810fe6257121aa5a96c67da1a122b2a8c8313f35`; 2927 bytes. The full transcript is attached as [full retained transcript](/api/content/media/c9773064-b40b-86e6-ac89-1689adeb3d67/file). Git archive was read directly from the named commit; the source modules and installer were separately hashed with `git show <commit>:<path>` bytes, giving exactly the three digests printed in this record. No source edits, credential access, service changes, live recovery or review approval formed part of this test run.

Historical transcript A (47 tests), preserved but unverified: original asserted command was the four-file node command above. Original asserted exit status was 0. Tested commit is unresolved: the introduction claimed a2cc60a157914459e5fd98e23bdc1f03430f6c27, while the log paragraph claimed f667d6aed53f6bac7e4fa0404e4ae784eabfa931. Neither assertion establishes the producing checkout. Digest of the exact transcript body retained here: 7a0075d7e37d5d14d505af9325d3ba48d7ecd9f8d0ba6efa0cdec773f6daa66c; 4795 UTF-8 bytes. The original external log digest assertion (1943c45c87e3bc123970c269feef7f5eb6689977c344a9e2804f12d8bf31fb54) is not treated as verification of this embedded transcript. This transcript does not verify the installed revision; the source-pinned fresh run above supersedes it for test applicability.

```text
✔ delegate pins Astra xhigh, disables tools and never gives Codex its Eve credential (2.694729ms)
✔ delegate off is idle and interrupted outcomes replay without a second model decision (3.49094ms)
✔ oversized evidence creates an explicit Elikem escalation instead of retrying forever (1.759531ms)
✔ clean fenced parking finishes a worker cycle; dirty or unproven parking preserves recovery (6.240341ms)
✔ operator memory budgets leave room for Codex and native allocations (2.167021ms)
✔ extra service headroom preserves host and service safety margins (0.384004ms)
✔ resource gates defer busy hosts and stop pressure before the kernel ceiling (0.225236ms)
✔ reviews require real evidence quotations, every check, and complete evidence (3.12326ms)
✔ the per-submission schema bounds every collection and rejects extra or oversized output (73.671451ms)
✔ citation diagnostics locate every mismatch without exposing model text or accepting it (0.72958ms)
✔ malformed output has safe specific diagnostics without an automatic approval or extra invocation (3.347685ms)
✔ document review reads the pinned source revision once and never follows current backlinks (1.605231ms)
✔ unavailable, substituted and malformed pinned sources cannot support approval (2.113029ms)
✔ source reads have a fixed request bound and explicit truncation without recursive fetching (0.674678ms)
✔ encoded evidence size and secret redaction are enforced before adding source text (3.169089ms)
✔ invalid or missing pinned relationship metadata stays explicit and makes no source calls (0.510414ms)
✔ a failed relationship read records missing evidence instead of stranding the review (0.462931ms)
✔ unattended review uses only Codex with no shell tools or Eve credentials (2.676662ms)
✔ long documents reach the model whole while code and JSON-encoded limits remain enforced (5.333236ms)
✔ oversized document discovery marks the excerpt and forbids automatic approval (1.981102ms)
✔ a reflowed citation gets one model repair against the unchanged exact evidence (1.929114ms)
✔ citation repair remains bounded and never relaxes validation or retries provider failures (1.859801ms)
✔ only one own technical report is requeued once, including named models; substantive outcomes are preserved (1.25747ms)
✔ an invalid repaired report stays unapproved with its model and safe failure reason (4.084838ms)
✔ R35.2: the reviewer lane runs each review as a session on its review claim (3.290009ms)
✔ recovery scans beyond recent missing-evidence reports and wraps its durable cursor (2.725215ms)
✔ both unattended roles do no model or repository work when their independent switch is off (0.897421ms)
✔ automatic worker preserves failed work and cannot silently move to another task (2.623594ms)
✔ missing Codex authentication prevents either unattended role from claiming work (0.520187ms)
✔ document application recovers a lost acknowledgement without applying twice (0.422162ms)
✔ turning the reviewer off still reconciles a human decision without starting another review (0.358976ms)
✔ a saved review completion resumes with its original request even after automation is off (0.705245ms)
✔ a successful automatic worker confirms its exact shipped commit reached origin/main (2.041161ms)
✔ R01.7: a task with an open question is never claimed; the next one without is (1.941783ms)
✔ R01.7: five open questions at the head of the queue do not hide the answered task behind them (3.304761ms)
✔ R01.7: an answered task behind more than 100 open questions is found past the cap and claimed (7.828907ms)
✔ R01.7: answered work is claimed with the questions and answers in its brief (1.559982ms)
✔ human decisions cannot be applied by the unattended reviewer (0.229373ms)
✔ common credential shapes are removed before model evidence is assembled (0.145609ms)
✔ code evidence reads only the shipped ancestor diff and an exact trusted receipt (84.14321ms)
✔ R25.2: a cancel requested on the lane row stops the run at the next heartbeat (18.700974ms)
✔ R35.2: Codex events map to typed activities, bounded; a run streams at most the cap (3.800081ms)
✔ recovery arguments require an explicit task and positive fencing token (1.763557ms)
✔ recovery refuses stale, replaced, foreign, finished or ambiguous work (3.452442ms)
✔ explicit recovery continues preserved changes without fetching, merging or claiming another task (4.543864ms)
✔ Codex child receives fixed bounded worker tool settings, never inherited NODE_OPTIONS (22.188979ms)
✔ live resource pressure aborts a resumed job and preserves its recovery state (8.896272ms)
ℹ tests 47
ℹ suites 0
ℹ pass 47
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 358.60202

```

Historical transcript B (46 tests), preserved but unverified: producing commit, command and process exit status were not recorded and remain unknown. A zero-failure summary is not evidence of process exit or source identity. Digest of the exact retained transcript body: 677da1f30b2760635ee1a314b3c2780c9601c77c030a25f1bc32c2bf03290244; 4686 UTF-8 bytes. No applicability to the installed revision is claimed; the source-pinned fresh run above supersedes this transcript.

```text
✔ delegate pins Astra xhigh, disables tools and never gives Codex its Eve credential (2.535068ms)
✔ delegate off is idle and interrupted outcomes replay without a second model decision (2.570812ms)
✔ oversized evidence creates an explicit Elikem escalation instead of retrying forever (1.65976ms)
✔ clean fenced parking finishes a worker cycle; dirty or unproven parking preserves recovery (5.86831ms)
✔ operator memory budgets leave room for Codex and native allocations (1.82679ms)
✔ extra service headroom preserves host and service safety margins (0.355902ms)
✔ resource gates defer busy hosts and stop pressure before the kernel ceiling (0.191767ms)
✔ reviews require real evidence quotations, every check, and complete evidence (3.353732ms)
✔ the per-submission schema bounds every collection and rejects extra or oversized output (72.87223ms)
✔ citation diagnostics locate every mismatch without exposing model text or accepting it (0.561449ms)
✔ malformed output has safe specific diagnostics without an automatic approval or extra invocation (5.725617ms)
✔ document review reads the pinned source revision once and never follows current backlinks (4.550481ms)
✔ unavailable, substituted and malformed pinned sources cannot support approval (3.615934ms)
✔ source reads have a fixed request bound and explicit truncation without recursive fetching (0.580928ms)
✔ encoded evidence size and secret redaction are enforced before adding source text (7.067852ms)
✔ invalid or missing pinned relationship metadata stays explicit and makes no source calls (0.389421ms)
✔ a failed relationship read records missing evidence instead of stranding the review (3.490857ms)
✔ unattended review uses only Codex with no shell tools or Eve credentials (2.417342ms)
✔ long documents reach the model whole while code and JSON-encoded limits remain enforced (12.706368ms)
✔ oversized document discovery marks the excerpt and forbids automatic approval (8.328987ms)
✔ a reflowed citation gets one model repair against the unchanged exact evidence (3.894069ms)
✔ citation repair remains bounded and never relaxes validation or retries provider failures (2.176513ms)
✔ only one own technical report is requeued once, including named models; substantive outcomes are preserved (2.251504ms)
✔ an invalid repaired report stays unapproved with its model and safe failure reason (5.905375ms)
✔ R35.2: the reviewer lane runs each review as a session on its review claim (3.358059ms)
✔ recovery scans beyond recent missing-evidence reports and wraps its durable cursor (1.532099ms)
✔ both unattended roles do no model or repository work when their independent switch is off (0.822879ms)
✔ automatic worker preserves failed work and cannot silently move to another task (7.711304ms)
✔ missing Codex authentication prevents either unattended role from claiming work (0.678404ms)
✔ document application recovers a lost acknowledgement without applying twice (0.505316ms)
✔ turning the reviewer off still reconciles a human decision without starting another review (0.449631ms)
✔ a saved review completion resumes with its original request even after automation is off (0.78349ms)
✔ a successful automatic worker confirms its exact shipped commit reached origin/main (2.050314ms)
✔ R01.7: a task with an open question is never claimed; the next one without is (5.073161ms)
✔ R01.7: five open questions at the head of the queue do not hide the answered task behind them (8.188117ms)
✔ R01.7: answered work is claimed with the questions and answers in its brief (1.449136ms)
✔ human decisions cannot be applied by the unattended reviewer (0.26711ms)
✔ common credential shapes are removed before model evidence is assembled (0.141351ms)
✔ code evidence reads only the shipped ancestor diff and an exact trusted receipt (102.495449ms)
✔ R25.2: a cancel requested on the lane row stops the run at the next heartbeat (32.769271ms)
✔ R35.2: Codex events map to typed activities, bounded; a run streams at most the cap (2.819944ms)
✔ recovery arguments require an explicit task and positive fencing token (1.988442ms)
✔ recovery refuses stale, replaced, foreign, finished or ambiguous work (7.361261ms)
✔ explicit recovery continues preserved changes without fetching, merging or claiming another task (6.695516ms)
✔ Codex child receives fixed bounded worker tool settings, never inherited NODE_OPTIONS (33.666691ms)
✔ live resource pressure aborts a resumed job and preserves its recovery state (7.786075ms)
ℹ tests 46
ℹ suites 0
ℹ pass 46
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 743.242253

```
