Disciplines · Decisions (ADRs)

ADR-0081: Eve watcher semantics before UI

but a schedule is not a watcher.

5sections8 minread

On this page
  • Status: accepted for Task 7.3 evidence
  • Date: 2026-09-09
  • Owners: Eve assistant runtime, Security, Operations, Privacy, Evaluation
  • Scope: Phase 7 Task 7.3 only

Context#

@oshun/assistant already contains a scheduler for one-shot and cron prompts, but a schedule is not a watcher. A watcher has a governed evidence source, a truth condition, evidence-age rules, repeated delivery semantics, revocable authority, privacy retention, and restart behavior. Exposing create/list/pause controls before these rules exist would make the UI the accidental source of truth and would invite incompatible event and polling behavior.

Task 7.3 therefore establishes a transport-free reference contract and state machine. It does not expose a Drawer tool, start a worker, call an external source, or send a notification.

Alternatives considered#

  • Extend the existing one-shot and cron prompt scheduler. Rejected: a schedule has no evidence source, truth condition, evidence age, delivery dedupe, revocable authority or retention policy. Adding conditions to it would turn a time trigger into an effect engine with no receipts.
  • Ship the Drawer tools first and settle semantics in each adapter. Rejected: the UI would become the accidental source of truth, and event and polling adapters would diverge (Context above).
  • Make an external workflow or job engine the watcher authority. Not chosen now: it adds infrastructure and would still need the same evidence, authority, privacy and dedupe rules on top. A transport-free contract keeps that choice open for Task 7.5's durable worker.
  • Exactly-once delivery through transactions with each destination. Rejected: destinations do not offer it. At-least-once delivery with a stable, subject-scoped delivery key passed as the destination's idempotency key is achievable everywhere.

Chosen: a pure state machine and store contract, with no transport, that every adapter calls.

Decision#

Source and condition#

A versioned WatcherDefinition is owned by an exact tenant and user and binds one purpose, source, condition, delivery reference, expiry, budgets, retries, and privacy policy.

Sources are a closed union:

  • An event source names an opaque source id, exact event type, and maximum evidence age. Event evidence carries an event id and durable source cursor.
  • A polling source additionally names a governed tool, five-field cron expression, IANA timezone, and explicit skip, latest, or bounded missed-run policy. A bounded policy retains at most the newest 100 due instants. This task selects already-derived due instants; the future worker owns cron enumeration and source invocation.

Conditions use a depth- and fanout-bounded declarative DSL over safe dotted payload paths. The only operators are strict scalar comparisons, string containment, existence, Boolean all/any, and not. No code, prompt, regular expression, or model judgment participates in condition truth.

Evidence and time#

Evidence must match the definition's source kind and source id; event evidence must also match the event type. It is fresh only inside the configured past-age window and a fixed 30-second future-skew allowance. An authorized stale or false record advances its source checkpoint so a poison record cannot replay forever. An authorization failure returns no transition and therefore cannot advance a checkpoint.

Quiet hours use local minutes in an explicitly validated IANA timezone. The next allowed delivery is found by walking real UTC instants and recomputing local wall time for each instant. This covers nonexistent spring-forward minutes and both occurrences of fall-back minutes instead of assuming a 24-hour local day. Quiet hours hold for every delivery claim, not only the first enqueue: a retry or a recovered entry that falls due inside them is moved to the first allowed instant (reason quiet_hours). Its operation deadline is not reset, so a deferral past the deadline ends the entry as a dead letter on its next claim (commit d531a8f1ec, with a regression test for a retry crossing into quiet hours and for recovery during them).

Delivery, dedupe, retry, and restart#

A true condition creates a durable outbox entry. Its SHA-256 delivery key binds the subject (tenant and user), watcher id/version, source id, and the source's stable trigger identity: event type/id or poll scheduled instant. The subject keeps two people's watchers that share an id, version, source and trigger distinct at a destination with one idempotency namespace (commit cc5b5f5106, 2026-10-01, with a test that a retry within one subject keeps its key). Mutable ingress-envelope ids, cursors, observation timestamps, and payload fields do not change that key, so a retried source event cannot evade dedupe. A separate digest binds the complete canonical evidence for audit. Delivery is at least once, not exactly once: a crash after the destination accepts a message but before the durable acknowledgement can cause redelivery. Every delivery adapter must pass the stable delivery key as its downstream idempotency key.

Dispatch follows this persistence sequence:

  1. Recheck current delivery authority.
  2. Claim one due outbox entry as dispatching and increment its attempt count.
  3. Persist that transition with the store's subject-scoped revision compare-and-set.
  4. Only after the compare-and-set wins, invoke the destination with the stable delivery key.
  5. Persist delivered, retryable pending, or terminal dead_letter state through another compare-and-set.

A process restart changes interrupted dispatching entries back to pending without changing their keys or attempt counts. Failed attempts use deterministic capped exponential backoff. The configured retry ceiling dead-letters an item, recovered work included: an entry whose attempts already reached the watcher's own ceiling (a crash during its last attempt) is dead-lettered instead of claimed again (commit de2e538490); the independent per-window attempt ceiling stops dispatch until the next budget window. Only bounded categorical transport error codes are retained.

Budgets and authority#

Each fixed window caps evaluations, unique deliveries, delivery attempts, tokens, and micro-USD cost. Usage is non-negative integer accounting supplied by the governed caller. Exhaustion suppresses new work and is recorded as a reason; a new window resets counters.

Authorization is fail-closed and is rechecked before every evidence evaluation, every delivery attempt, and every management or erasure command. The grant must match the exact tenant, user, purpose, source, governed polling tool when applicable, delivery target when delivering, requested action, and current time. The definition stores opaque registry references, never credentials or raw webhook URLs.

Lifecycle, persistence, and privacy#

Lifecycle is active, paused, cancelled, or expired. Pause/resume is an authorized reversible transition. Cancellation is terminal. Expiry applies to active and paused watchers and prevents further evaluation or delivery.

The persistence key is (tenant_id, user_id, watcher_id), so the same opaque watcher id in another subject is distinct. Initial creation refuses a conflict instead of overwriting an existing revision. Updates use an advancing revision and compare-and-set to reject stale workers. Reads, listings, and erasure use the exact tenant/user predicates. The process-local reference store can export and restore a snapshot; the Postgres port retains definition, state, outbox, checkpoint, budgets, and revision as durable JSONB.

Evidence receipts retain only scalar leaves at policy-allowlisted payload paths; allowlisting a parent object does not retain its unlisted descendants. Stale or budget-rejected evidence retains no payload, whether the evaluation budget or the delivery budget rejected it (commit de2e538490). Delivery metadata retention must cover the full evidence-age window plus future skew, so a still-acceptable replay cannot outlive its dedupe key. Retention windows purge evidence and settled delivery metadata; pending and dispatching work is not silently deleted. Cancellation (with eraseContentOnCancel) and an authorized subject erasure remove evidence receipts and every queued (pending) outbox entry at once; erasure also removes suppressions and checkpoints. An entry already dispatching has been handed to the external channel and is treated separately. It stays as a minimal outcome receipt (delivery key, status, attempts, times and error code, with no payload and no trace envelope), marked so it is never retried, until it settles as delivered or dead-lettered. A restart does not change that: recovery turns an interrupted receipt of this kind into a dead letter with an unknown outcome, never back into pending work, and delivery never claims a retry-suppressed entry (commit de2e538490). The external write may therefore complete after cancellation or erasure. Removing the receipt could not recall it, only make the outcome unknowable. Storage deletion of the subject (deleteSubject, or deleteUser for an account) is different: it removes the whole watcher row, receipts included, and fences later writes for that subject. Both stores keep the fence: PostgresWatcherStore in its deleted-subjects and deleted-users tables, and InMemoryWatcherStore in a deleted-subjects set carried in its snapshot, so a stale write after deletion returns false, even after a restart (commit d70af90fa0, with a regression test). A settlement arriving after it has nothing to update and is not recorded, though the external write may still have happened.

Verification (2026-10-01)#

At the commit that added quiet-hours enforcement (d531a8f1ec), from libs/oshun/assistant:

  • npx vitest run src/watchers/ src/loops/: 108 passed, 5 skipped. The 5 skipped are the loop store's and board-event reader's PostgreSQL integration tests, which need a test database. Every watcher test ran, including the regression tests for each fix named above.
  • npx tsc --noEmit -p tsconfig.lib.json: no errors.
  • npx eslint src/watchers/watcher-engine.ts: no problems.

Consequences and honest boundary#

The repository now has one testable semantic authority for watcher definitions, evaluation, delivery state, recovery, tenancy, and privacy. Event and polling workers can share the same state transitions instead of inventing behavior in their adapters. The design deliberately exposes no unbounded code execution or destination credential.

This is a source/static and process-local/SQL-boundary contract, not live watcher reliability evidence. It does not enumerate cron fires, poll a real source, consume a real event stream, send a notification, prove a real database restart, or exercise a distributed worker race. It does not cite evidence in a user-facing notification. Task 7.4 owns gated product tools and notification shape; Task 7.5 owns multiple live watcher classes, duplicate/flapping/outage behavior, restart recovery, poisoned inputs, revoked permissions, and cancellation under realistic service conditions. Task 7.3 therefore closes only the pre-UI semantics contract; Phase 7 and G3/G11-G18 remain open.