- Status: accepted for Task 7.3 evidence
- Date: 2026-09-09
- Owners: Eve assistant runtime, Security, Operations, Privacy, Evaluation
- Scope: Phase 7 Task 7.3 only
Context#
@oshun/assistant already contains a scheduler for one-shot and cron prompts,
but a schedule is not a watcher. A watcher has a governed evidence source, a
truth condition, evidence-age rules, repeated delivery semantics, revocable
authority, privacy retention, and restart behavior. Exposing create/list/pause
controls before these rules exist would make the UI the accidental source of
truth and would invite incompatible event and polling behavior.
Task 7.3 therefore establishes a transport-free reference contract and state machine. It does not expose a Drawer tool, start a worker, call an external source, or send a notification.
Alternatives considered#
- Extend the existing one-shot and cron prompt scheduler. Rejected: a schedule has no evidence source, truth condition, evidence age, delivery dedupe, revocable authority or retention policy. Adding conditions to it would turn a time trigger into an effect engine with no receipts.
- Ship the Drawer tools first and settle semantics in each adapter. Rejected: the UI would become the accidental source of truth, and event and polling adapters would diverge (Context above).
- Make an external workflow or job engine the watcher authority. Not chosen now: it adds infrastructure and would still need the same evidence, authority, privacy and dedupe rules on top. A transport-free contract keeps that choice open for Task 7.5's durable worker.
- Exactly-once delivery through transactions with each destination. Rejected: destinations do not offer it. At-least-once delivery with a stable, subject-scoped delivery key passed as the destination's idempotency key is achievable everywhere.
Chosen: a pure state machine and store contract, with no transport, that every adapter calls.
Decision#
Source and condition#
A versioned WatcherDefinition is owned by an exact tenant and user and binds
one purpose, source, condition, delivery reference, expiry, budgets, retries,
and privacy policy.
Sources are a closed union:
- An event source names an opaque source id, exact event type, and maximum evidence age. Event evidence carries an event id and durable source cursor.
- A polling source additionally names a governed tool, five-field cron
expression, IANA timezone, and explicit
skip,latest, or bounded missed-run policy. A bounded policy retains at most the newest 100 due instants. This task selects already-derived due instants; the future worker owns cron enumeration and source invocation.
Conditions use a depth- and fanout-bounded declarative DSL over safe dotted payload paths. The only operators are strict scalar comparisons, string containment, existence, Boolean all/any, and not. No code, prompt, regular expression, or model judgment participates in condition truth.
Evidence and time#
Evidence must match the definition's source kind and source id; event evidence must also match the event type. It is fresh only inside the configured past-age window and a fixed 30-second future-skew allowance. An authorized stale or false record advances its source checkpoint so a poison record cannot replay forever. An authorization failure returns no transition and therefore cannot advance a checkpoint.
Quiet hours use local minutes in an explicitly validated IANA timezone. The next
allowed delivery is found by walking real UTC instants and recomputing local
wall time for each instant. This covers nonexistent spring-forward minutes and
both occurrences of fall-back minutes instead of assuming a 24-hour local day.
Quiet hours hold for every delivery claim, not only the first enqueue: a retry
or a recovered entry that falls due inside them is moved to the first allowed
instant (reason quiet_hours). Its operation deadline is not reset, so a
deferral past the deadline ends the entry as a dead letter on its next claim
(commit d531a8f1ec, with a regression test for a retry crossing into quiet
hours and for recovery during them).
Delivery, dedupe, retry, and restart#
A true condition creates a durable outbox entry. Its SHA-256 delivery key binds the subject (tenant and user), watcher id/version, source id, and the source's stable trigger identity: event type/id or poll scheduled instant. The subject keeps two people's watchers that share an id, version, source and trigger distinct at a destination with one idempotency namespace (commit cc5b5f5106, 2026-10-01, with a test that a retry within one subject keeps its key). Mutable ingress-envelope ids, cursors, observation timestamps, and payload fields do not change that key, so a retried source event cannot evade dedupe. A separate digest binds the complete canonical evidence for audit. Delivery is at least once, not exactly once: a crash after the destination accepts a message but before the durable acknowledgement can cause redelivery. Every delivery adapter must pass the stable delivery key as its downstream idempotency key.
Dispatch follows this persistence sequence:
- Recheck current delivery authority.
- Claim one due outbox entry as
dispatchingand increment its attempt count. - Persist that transition with the store's subject-scoped revision compare-and-set.
- Only after the compare-and-set wins, invoke the destination with the stable delivery key.
- Persist
delivered, retryablepending, or terminaldead_letterstate through another compare-and-set.
A process restart changes interrupted dispatching entries back to pending
without changing their keys or attempt counts. Failed attempts use deterministic
capped exponential backoff. The configured retry ceiling dead-letters an item, recovered work included: an
entry whose attempts already reached the watcher's own ceiling (a crash during
its last attempt) is dead-lettered instead of claimed again (commit de2e538490);
the independent per-window attempt ceiling stops dispatch until the next budget
window. Only bounded categorical transport error codes are retained.
Budgets and authority#
Each fixed window caps evaluations, unique deliveries, delivery attempts, tokens, and micro-USD cost. Usage is non-negative integer accounting supplied by the governed caller. Exhaustion suppresses new work and is recorded as a reason; a new window resets counters.
Authorization is fail-closed and is rechecked before every evidence evaluation, every delivery attempt, and every management or erasure command. The grant must match the exact tenant, user, purpose, source, governed polling tool when applicable, delivery target when delivering, requested action, and current time. The definition stores opaque registry references, never credentials or raw webhook URLs.
Lifecycle, persistence, and privacy#
Lifecycle is active, paused, cancelled, or expired. Pause/resume is an
authorized reversible transition. Cancellation is terminal. Expiry applies to
active and paused watchers and prevents further evaluation or delivery.
The persistence key is (tenant_id, user_id, watcher_id), so the same opaque
watcher id in another subject is distinct. Initial creation refuses a conflict
instead of overwriting an existing revision. Updates use an advancing revision
and compare-and-set to reject stale workers. Reads, listings, and erasure use
the exact tenant/user predicates. The process-local reference store can export
and restore a snapshot; the Postgres port retains definition, state, outbox,
checkpoint, budgets, and revision as durable JSONB.
Evidence receipts retain only scalar leaves at policy-allowlisted payload paths;
allowlisting a parent object does not retain its unlisted descendants. Stale or
budget-rejected evidence retains no payload, whether the evaluation budget or
the delivery budget rejected it (commit de2e538490). Delivery metadata retention must
cover the full evidence-age window plus future skew, so a still-acceptable
replay cannot outlive its dedupe key. Retention windows purge evidence and
settled delivery metadata; pending and dispatching work is not silently deleted.
Cancellation (with eraseContentOnCancel) and an authorized subject erasure
remove evidence receipts and every queued (pending) outbox entry at once;
erasure also removes suppressions and checkpoints. An entry already
dispatching has been handed to the external channel and is treated
separately. It stays as a minimal outcome receipt (delivery key, status,
attempts, times and error code, with no payload and no trace envelope), marked
so it is never retried, until it settles as delivered or dead-lettered. A
restart does not change that: recovery turns an interrupted receipt of this kind
into a dead letter with an unknown outcome, never back into pending work, and
delivery never claims a retry-suppressed entry (commit de2e538490). The
external write may therefore complete after cancellation or erasure. Removing
the receipt could not recall it, only make the outcome unknowable. Storage
deletion of the subject (deleteSubject, or deleteUser for an account) is
different: it removes the whole watcher row, receipts included, and fences
later writes for that subject. Both stores keep the fence:
PostgresWatcherStore in its deleted-subjects and deleted-users tables, and
InMemoryWatcherStore in a deleted-subjects set carried in its snapshot, so a
stale write after deletion returns false, even after a restart (commit
d70af90fa0, with a regression test). A settlement arriving after it has nothing to
update and is not recorded, though the external write may still have
happened.
Verification (2026-10-01)#
At the commit that added quiet-hours enforcement (d531a8f1ec), from
libs/oshun/assistant:
npx vitest run src/watchers/ src/loops/: 108 passed, 5 skipped. The 5 skipped are the loop store's and board-event reader's PostgreSQL integration tests, which need a test database. Every watcher test ran, including the regression tests for each fix named above.npx tsc --noEmit -p tsconfig.lib.json: no errors.npx eslint src/watchers/watcher-engine.ts: no problems.
Consequences and honest boundary#
The repository now has one testable semantic authority for watcher definitions, evaluation, delivery state, recovery, tenancy, and privacy. Event and polling workers can share the same state transitions instead of inventing behavior in their adapters. The design deliberately exposes no unbounded code execution or destination credential.
This is a source/static and process-local/SQL-boundary contract, not live watcher reliability evidence. It does not enumerate cron fires, poll a real source, consume a real event stream, send a notification, prove a real database restart, or exercise a distributed worker race. It does not cite evidence in a user-facing notification. Task 7.4 owns gated product tools and notification shape; Task 7.5 owns multiple live watcher classes, duplicate/flapping/outage behavior, restart recovery, poisoned inputs, revoked permissions, and cancellation under realistic service conditions. Task 7.3 therefore closes only the pre-UI semantics contract; Phase 7 and G3/G11-G18 remain open.