# Eve Operations, Observability, and Maintenance

Operating Eve means keeping identity, policy, provider, tools, persistence,
evaluation, and UI disclosures aligned. A green HTTP health check is necessary
but cannot prove that routing, grounding, confirmations, or model quality are
healthy.

```mermaid
stateDiagram-v2
  [*] --> Healthy
  Healthy --> Degraded: dependency quality or telemetry signal breaches
  Degraded --> ReadOnly: mutation path cannot be proven safe
  Degraded --> Fallback: measured compatible provider or deterministic path
  Degraded --> Unavailable: no honest safe fallback
  ReadOnly --> Recovering: dependency and authoritative state restored
  Fallback --> Recovering: primary path passes probes and canary
  Unavailable --> Recovering: configuration service or provider repaired
  Recovering --> Healthy: routing grounding receipts and evals reconcile
  Recovering --> Degraded: verification or reconciliation fails
  Healthy --> Incident: tenant safety privacy or false-effect signal
  Degraded --> Incident: blast radius or control breach confirmed
  Incident --> Contained
  Contained --> Recovering
```

Recovery means the assistant's end-to-end promises reconcile—identity, policy,
routing, tools, grounding, disclosures, storage, and evaluation—not merely that
one health endpoint returned green.

## Operational dependency map

| Dependency                         | Operational concern                               | Required degraded behavior                                                   |
| ---------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------- |
| Identity/session and tenant policy | Authentication, role, tenant, release scope.      | Deny or reduce to public/general behavior; never cross tenants.              |
| Model provider and registry        | Availability, pinned model/settings, rate limits. | Deterministic/templated fallback or explicit unavailable state.              |
| Iris memory                        | Recall/write, consent, retention, DSAR.           | Continue without durable memory and show the actual memory posture.          |
| Sophia/docs retrieval              | Evidence and documentation corpus freshness.      | Label unsupported/ungrounded output; do not invent citations.                |
| Tool services                      | Read/action capability and receipts.              | Report unavailable/partial/timeout with no false completion.                 |
| Conversation store                 | Durable history and reconnect.                    | Preserve turn integrity, avoid duplicate mutations, explain continuity loss. |
| Workbench store/MCP                | Proposals, confirmations, queue, leases, reports. | Refuse mutation or queue work safely; keep authoritative state inspectable.  |
| Voice providers/browser APIs       | STT/TTS and media permissions.                    | Text-first operation with explicit voice unavailable/permission state.       |
| Telemetry/evaluation pipeline      | Quality, cost, routing, refusal, crash signals.   | Do not block safe user response by default; alert on blind operation.        |

## Configuration and secrets

Provider endpoints, model bindings, limits, feature flags, release scopes, tool
availability, and kill switches are validated at startup and redacted in logs.
The committed environment reference in the Docs Center is the canonical map of
configuration names; secrets belong in the platform secret store, not prompts,
tool descriptions, client bundles, or evidence attachments.

Changing a model name or provider setting is a release-affecting registry
change. Changing a prompt, skill, tool description, or eval dependency follows
the quality process in
[Models, quality, and evaluation](./models-quality-and-evaluation.md).

## Telemetry schema

Useful turn telemetry records stable identifiers and bounded enums rather than
raw sensitive content:

- request/turn/session identifiers and plane;
- tenant/workspace hash or authorized dimension;
- persona and release scope;
- task family, router reason, and skill/toolset versions;
- model/provider/leg, cached and uncached tokens, latency, and cost;
- tool names, typed outcome classes, and confirmation/lease identifiers;
- grounding/support state and checker result;
- refusal/safety-supersession category;
- stream/reconnect/crash outcome;
- user feedback and linked evaluation/defect identifiers.

Prompts, selections, transcripts, retrieved documents, and tool payloads require
separate privacy review and redaction. Observability should answer what failed
without becoming a shadow memory store.

## Health and service-level signals

Monitor at least:

- invocation-to-first-render and turn-to-first-sentence latency;
- total turn latency by family and provider leg;
- route distribution, unknown/misroute audit rate, and family fallback rate;
- tool call success, timeout, denial, invalid-argument, and partial outcomes;
- confirmation proposal-to-decision and stale/replay failures;
- grounding holds, unsupported claim rate, and source-corpus freshness;
- refusal and safety-supersession rate by policy version;
- provider escalation, retry, budget-exhaustion, and kill-switch state;
- conversation persistence/reconnect failure and duplicate-operation prevention;
- crash ingest and client voice/tour failure modes;
- per-family eval floors and prompt-hash drift;
- cost per successful outcome, not merely cost per call.

Alert thresholds should link to a runbook and identify whether member, builder,
agent, or quality plane is affected.

## Incident triage

1. Establish blast radius by plane, persona, tenant, task family, provider,
   release, and tool.
2. Check hard safety/authorization invariants before quality symptoms.
3. Inspect recent model registry, prompt/skill/tool-description, router,
   contract, and policy changes.
4. Reproduce with a redacted trace or the smallest matching eval/battery case.
5. Use the narrowest kill switch, model rollback, tool disable, or route clamp
   that restores a safe state.
6. Verify user-visible degraded copy and confirmation/persistence integrity.
7. Add the failure to the eval/defect ledger before closing the incident.
8. Restore gradually and record the version/evidence that cleared the gate.

If the incident touches memory, rights, crisis handling, or cross-tenant access,
follow the relevant privacy/security incident process in addition to the
assistant runbook.

## Provider degradation

Eve should expose the capability that remains, not simulate the missing one.
Examples:

- model unavailable: deterministic safe response or bounded retry/escalation;
- retrieval unavailable: no grounded badge or fabricated citation;
- read tool unavailable: state that live workspace status could not be checked;
- mutation service unavailable: do not render a confirmation as if it were
  persisted;
- TTS unavailable: keep text and announce voice unavailability;
- telemetry degraded: operate only within the documented blind-mode policy and
  raise an operational alert.

## Evaluation and release procedure

Before releasing an assistant-affecting change, run the narrow unit/integration
tests for the changed stage, the relevant family deck and prompt ratchet, the
affected browser/mobile journey, and Docs Center checks when contracts or
behavior changed. High-risk changes also require adversarial, safety, grounded
answer, persistence, and release-gate suites.

The current surface-polish closure record is
[Eve Deep Test and Polish](../../../EVE_DEEP_TEST_AND_POLISH_TODOS_2026-08-04.md);
the ongoing measured loop is
[Eve SMX maintenance](../../../docs/agents/eve-smx-maintenance.md).

## Routine maintenance

**Daily/continuous:** provider health, hard-invariant alerts, budget/kill
switch, crash/error spikes, queue/lease hygiene, corpus freshness, and
confirmation failures.

**Weekly:** SMX decks and prompt ratchets, misroute/refusal review, cost and
latency report, judge drift, new defect-to-case intake, stale confirmations and
agent leases.

**Per release:** model/skill/policy/tool registry diff, migrations/contracts,
tenant/release scope, accessibility and mobile flows, crisis/memory/grounding
journeys, operational rollback rehearsal.

## Related

- [Safety, governance, and privacy](./safety-governance-and-privacy.md)
- [Models, quality, and evaluation](./models-quality-and-evaluation.md)
- [Observability and quality architecture](../../architecture/observability-and-quality.md)
- [Platform foundations and security](../../features/platform-foundations-and-security.md)
- [Eve SMX maintenance](../../../docs/agents/eve-smx-maintenance.md)