Operating Eve means keeping identity, policy, provider, tools, persistence, evaluation, and UI disclosures aligned. A green HTTP health check is necessary but cannot prove that routing, grounding, confirmations, or model quality are healthy.
Recovery means the assistant's end-to-end promises reconcile—identity, policy, routing, tools, grounding, disclosures, storage, and evaluation—not merely that one health endpoint returned green.
Operational dependency map#
| Dependency | Operational concern | Required degraded behavior |
|---|---|---|
| Identity/session and tenant policy | Authentication, role, tenant, release scope. | Deny or reduce to public/general behavior; never cross tenants. |
| Model provider and registry | Availability, pinned model/settings, rate limits. | Deterministic/templated fallback or explicit unavailable state. |
| Iris memory | Recall/write, consent, retention, DSAR. | Continue without durable memory and show the actual memory posture. |
| Sophia/docs retrieval | Evidence and documentation corpus freshness. | Label unsupported/ungrounded output; do not invent citations. |
| Tool services | Read/action capability and receipts. | Report unavailable/partial/timeout with no false completion. |
| Conversation store | Durable history and reconnect. | Preserve turn integrity, avoid duplicate mutations, explain continuity loss. |
| Workbench store/MCP | Proposals, confirmations, queue, leases, reports. | Refuse mutation or queue work safely; keep authoritative state inspectable. |
| Voice providers/browser APIs | STT/TTS and media permissions. | Text-first operation with explicit voice unavailable/permission state. |
| Telemetry/evaluation pipeline | Quality, cost, routing, refusal, crash signals. | Do not block safe user response by default; alert on blind operation. |
Configuration and secrets#
Provider endpoints, model bindings, limits, feature flags, release scopes, tool availability, and kill switches are validated at startup and redacted in logs. The committed environment reference in the Docs Center is the canonical map of configuration names; secrets belong in the platform secret store, not prompts, tool descriptions, client bundles, or evidence attachments.
Changing a model name or provider setting is a release-affecting registry change. Changing a prompt, skill, tool description, or eval dependency follows the quality process in Models, quality, and evaluation.
Telemetry schema#
Useful turn telemetry records stable identifiers and bounded enums rather than raw sensitive content:
- request/turn/session identifiers and plane;
- tenant/workspace hash or authorized dimension;
- persona and release scope;
- task family, router reason, and skill/toolset versions;
- model/provider/leg, cached and uncached tokens, latency, and cost;
- tool names, typed outcome classes, and confirmation/lease identifiers;
- grounding/support state and checker result;
- refusal/safety-supersession category;
- stream/reconnect/crash outcome;
- user feedback and linked evaluation/defect identifiers.
Prompts, selections, transcripts, retrieved documents, and tool payloads require separate privacy review and redaction. Observability should answer what failed without becoming a shadow memory store.
Health and service-level signals#
Monitor at least:
- invocation-to-first-render and turn-to-first-sentence latency;
- total turn latency by family and provider leg;
- route distribution, unknown/misroute audit rate, and family fallback rate;
- tool call success, timeout, denial, invalid-argument, and partial outcomes;
- confirmation proposal-to-decision and stale/replay failures;
- grounding holds, unsupported claim rate, and source-corpus freshness;
- refusal and safety-supersession rate by policy version;
- provider escalation, retry, budget-exhaustion, and kill-switch state;
- conversation persistence/reconnect failure and duplicate-operation prevention;
- crash ingest and client voice/tour failure modes;
- per-family eval floors and prompt-hash drift;
- cost per successful outcome, not merely cost per call.
Alert thresholds should link to a runbook and identify whether member, builder, agent, or quality plane is affected.
Incident triage#
- Establish blast radius by plane, persona, tenant, task family, provider, release, and tool.
- Check hard safety/authorization invariants before quality symptoms.
- Inspect recent model registry, prompt/skill/tool-description, router, contract, and policy changes.
- Reproduce with a redacted trace or the smallest matching eval/battery case.
- Use the narrowest kill switch, model rollback, tool disable, or route clamp that restores a safe state.
- Verify user-visible degraded copy and confirmation/persistence integrity.
- Add the failure to the eval/defect ledger before closing the incident.
- Restore gradually and record the version/evidence that cleared the gate.
If the incident touches memory, rights, crisis handling, or cross-tenant access, follow the relevant privacy/security incident process in addition to the assistant runbook.
Provider degradation#
Eve should expose the capability that remains, not simulate the missing one. Examples:
- model unavailable: deterministic safe response or bounded retry/escalation;
- retrieval unavailable: no grounded badge or fabricated citation;
- read tool unavailable: state that live workspace status could not be checked;
- mutation service unavailable: do not render a confirmation as if it were persisted;
- TTS unavailable: keep text and announce voice unavailability;
- telemetry degraded: operate only within the documented blind-mode policy and raise an operational alert.
Evaluation and release procedure#
Before releasing an assistant-affecting change, run the narrow unit/integration tests for the changed stage, the relevant family deck and prompt ratchet, the affected browser/mobile journey, and Docs Center checks when contracts or behavior changed. High-risk changes also require adversarial, safety, grounded answer, persistence, and release-gate suites.
The current surface-polish closure record is Eve Deep Test and Polish; the ongoing measured loop is Eve SMX maintenance.
Routine maintenance#
Daily/continuous: provider health, hard-invariant alerts, budget/kill switch, crash/error spikes, queue/lease hygiene, corpus freshness, and confirmation failures.
Weekly: SMX decks and prompt ratchets, misroute/refusal review, cost and latency report, judge drift, new defect-to-case intake, stale confirmations and agent leases.
Per release: model/skill/policy/tool registry diff, migrations/contracts, tenant/release scope, accessibility and mobile flows, crisis/memory/grounding journeys, operational rollback rehearsal.