# Workbench Operations, Observability, and Release

Workbench operations cover interactive shells, APIs, workers, queues, databases,
object storage, search/graph projections, media/model providers, DCC/engine
bridges, and external delivery. A green route alone is not workbench health; the
supported journey and its recovery path must be observable end to end.

```mermaid
stateDiagram-v2
  [*] --> Ready
  Ready --> Degraded: dependency or SLO breach
  Degraded --> ReadOnly: writes cannot be proven safe
  Degraded --> Queuing: durable bounded backlog is safe
  Degraded --> Unavailable: no honest capability remains
  ReadOnly --> Recovering: stores and projections reconcile
  Queuing --> Recovering: workers providers and fences recover
  Unavailable --> Recovering: configuration or dependency restored
  Recovering --> Ready: journeys receipts and invariants pass
  Recovering --> Degraded: validation or capacity fails
  Ready --> Incident: isolation rights safety or external-effect breach
  Degraded --> Incident: blast radius becomes material
  Incident --> Contained
  Contained --> Recovering
```

Readiness is a capability state. A shell can remain available while a mutating,
provider-backed, or export path is read-only, queued, or explicitly unavailable.

## Operational inventory

For each domain, maintain:

- supported UI/native/CLI entry points and deployment units;
- APIs, event topics, queues, scheduled cycles, workers, remote hosts, and
  callbacks;
- databases/schemas, blob stores, caches, search/vector/graph indices, audit and
  signing stores;
- provider/model/tool/DCC dependencies and credentials;
- feature flags, tenant scopes, configuration, secrets, limits, and ownership;
- dashboards, alerts, runbooks, recovery objectives, backups, restore targets,
  game-day scenarios, and release gates.

The generic
[estate inventory](../../../evidence/v1-workbenches/inventory/README.md) and
domain inventories seed this register. Deployment discovery must reconcile them
with what actually runs.

## Health and readiness

**Liveness** answers whether the process can make progress. **Readiness**
answers whether it can safely accept its supported traffic. Readiness includes
critical migrations, stores, queue/outbox, identity/policy configuration,
required providers, and incompatible version fences. A noncritical provider may
yield a declared degraded-ready state only when capability-specific degradation
is safe and visible.

Workers expose lease/fencing health, backlog age/depth, attempts, poison/dead
letters, throughput, latency, and result/reclamation state. DCC/remote hosts add
host identity, supported versions/capabilities, project scope, heartbeat,
current lease, artifact transfer, and command/result receipts.

## Service-level indicators

Measure user and control outcomes by domain and capability:

| Signal            | Examples                                                                                                                                             |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Availability      | Authorized project/source/session opens; command acceptance; evidence/history load; job status; export/download; remote-host reachability.           |
| Latency           | Interactive read/write, search, save acknowledgement, job queue-to-start, processing phase, provider/DCC round trip, projection lag, notification.   |
| Correctness       | Schema and invariant failures, conflict rate, duplicate effects, stale projections, result validation, lineage completeness, signature verification. |
| Safety and rights | Denials by reason, cross-tenant probes, quarantine, policy/rights blocks, expiry/deletion backlog, export exclusions, provider/model withdrawal.     |
| Reliability       | Retry success, poison jobs, lease loss, outbox lag, restore consistency, callback/update dedupe, cancellation/compensation, crash loops.             |
| Experience        | UI error/partial/offline state, save honesty, accessibility regressions, unsupported formats/versions, time to recover or complete a core journey.   |

Metrics use bounded dimensions; tenant/resource ids and content do not become
high-cardinality labels. Correlation and trace ids link sanitized logs, traces,
audit, job, provider, and UI errors without exposing secrets.

## Degradation

Every dependency has a capability plan:

- what reads remain available and whether they are stale;
- what writes are blocked, queued, or safely local;
- whether jobs can start, continue, publish, retry, or cancel;
- what provider/model/tool fallback is compatible and authorized;
- what the user sees and what operators are alerted on;
- how the system detects recovery and reconciles queued/stale state;
- which data or results are withheld rather than guessed.

Fallback quality is verified for the same contract. Silently changing a model,
renderer, codec, DCC version, or evaluation method can invalidate comparison and
provenance even when a request succeeds.

## Capacity and resource safety

Plan limits for upload/ingest, media duration/resolution, archive expansion,
projects/sources/anchors, concurrent sessions, job queue/backlog, GPU/CPU/
memory/disk, database connections/locks, blob bandwidth, search/index growth,
provider quotas, exports, and remote hosts.

Admission control distinguishes user, tenant, capability, and global capacity.
Backpressure is end to end; accepting work into an unbounded local queue is not
protection. Processing sandboxes have time, memory, CPU, output, recursion, and
network limits. Capacity failures yield explicit retry/limit guidance and do not
corrupt source or history.

## Backup, restore, and disaster recovery

Recovery covers system-of-record relational data, blobs, revision/history,
outbox/inbox and job state, audit/signing keys and evidence, plus the sources
needed to rebuild search/vector/graph projections. External providers and DCC
hosts require reconciliation because a database restore cannot rewind them.

A valid restore proves:

- tenant/resource counts and sampled semantic invariants;
- database-to-blob and source-to-derived reachability;
- history ancestry and no impossible job/lease state;
- outbox/inbox idempotency and external-effect reconciliation;
- projection rebuild and cutover consistency;
- rights, holds, expiry, deletion/tombstone, and revoked-access state;
- signature verification and key availability;
- application journeys, not only store health.

The
[Yemaya backup/restore runbook](../../../docs/runbooks/yemaya-study-backup-restore.md)
is the domain's worked example. Shared restore classes and reference edges are
implemented in `libs/oshun/workbench-kit/src/restore-consistency.ts`.

## Rollout and rollback

Roll out in explicit rings or cohorts with owner, entry criteria, measurement
window, abort conditions, and promotion decision. Flags are tenant/capability
scoped and observable. Schema, contract, worker, projection, client, and
provider versions use a compatibility window; mixed-version behavior is tested.

Rollback is safe only if the previous version can interpret newly written
records and external effects. Otherwise use forward-fix, disable new writes,
keep compatible readers, or compensate. Never destroy migration or audit
evidence to make the old build start.

Each release records source revision, generated artifacts, migrations,
configuration/flags, dependency/model/tool versions, test and evaluation
evidence, security/privacy review, rollout state, and approver.

## Incident handling

The common sequence is:

1. Detect and declare the affected capability, tenants, data, and time window.
2. Contain unsafe writes, providers, workers, routes, remote hosts, keys, or
   exports while retaining supported safe reads where possible.
3. Preserve sanitized evidence and audit; do not copy sensitive content into an
   incident channel.
4. Establish source of truth, blast radius, irreversible external effects, and
   deletion/rights implications.
5. Repair, replay/rebuild/reconcile with idempotency and version fences.
6. Verify the user journey, tenant isolation, data/provenance, and recovery
   objective before reopening.
7. Communicate impact and residual risk; update runbooks, alerts, gates, and
   game-day coverage.

## Domain-specific operational focus

- **Isis:** GPU/provider capacity, model/workflow compatibility, queue and
  output lineage, safety/policy, consistency/parity, and generated asset
  reclamation.
- **Metis:** learning/assessment availability, evaluator/rubric versioning,
  progress/event ordering, credentials, integrations, and learner-data privacy.
- **Yemaya:** large media ingest/processing, parsers, analysis providers,
  projection/search/graph, rights expiry, evidence/export, deletion, and
  cross-domain adapters.
- **Veritas:** confidential sources, research/claim state, review/embargo,
  schedules, publish/delivery/correction/takedown, and social provider limits.
- **Euterpe:** realtime audio latency/glitches, collaboration consistency,
  native/browser/plugin compatibility, recording durability, and export.
- **Aja:** capture/reference ingest, motion jobs/models, skeleton/format
  compatibility, performer rights, large derived artifacts, and pipeline
  delivery.
- **Bellona:** bridge/host connectivity, engine/tool versions, remote command
  authorization, build/render queues, artifact transfer, approvals, and
  external-effect reconciliation.

## Release evidence checklist

- Targeted lint, type, unit, contract/schema, migration, and integration gates.
- Core browser/native/CLI journeys with success, denial, error, retry, partial,
  offline, conflict, cancellation, and recovery states.
- Accessibility and performance budgets for affected compositions.
- Security, tenant, rights, export, deletion, provider, and external-tool tests.
- Current dashboards/alerts/runbooks and a verified restore/rollback path when
  state or effects changed.
- Inventory and generated artifact freshness.
- Deployed smoke/canary and promotion/abort evidence.

Do not mark a capability complete because one package test passed. Completion
means the supported entry point, real effect, enforcement, observability,
failure handling, and recovery evidence agree.