Workbench operations cover interactive shells, APIs, workers, queues, databases, object storage, search/graph projections, media/model providers, DCC/engine bridges, and external delivery. A green route alone is not workbench health; the supported journey and its recovery path must be observable end to end.
This picture is wider than it can be shown here. Move across it, or read it as text.
Drawn from its Mermaid source.
As text
Parts: - Start (initial) - Ready - Degraded - ReadOnly - Queuing - Unavailable - Recovering - Incident - Contained Connections: - Start to Ready - Ready to Degraded: dependency or SLO breach - Degraded to ReadOnly: writes cannot be proven safe - Degraded to Queuing: durable bounded backlog is safe - Degraded to Unavailable: no honest capability remains - ReadOnly to Recovering: stores and projections reconcile - Queuing to Recovering: workers providers and fences recover - Unavailable to Recovering: configuration or dependency restored - Recovering to Ready: journeys receipts and invariants pass - Recovering to Degraded: validation or capacity fails - Ready to Incident: isolation rights safety or external-effect breach - Degraded to Incident: blast radius becomes material - Incident to Contained - Contained to Recovering
Show source
stateDiagram-v2
[*] --> Ready
Ready --> Degraded: dependency or SLO breach
Degraded --> ReadOnly: writes cannot be proven safe
Degraded --> Queuing: durable bounded backlog is safe
Degraded --> Unavailable: no honest capability remains
ReadOnly --> Recovering: stores and projections reconcile
Queuing --> Recovering: workers providers and fences recover
Unavailable --> Recovering: configuration or dependency restored
Recovering --> Ready: journeys receipts and invariants pass
Recovering --> Degraded: validation or capacity fails
Ready --> Incident: isolation rights safety or external-effect breach
Degraded --> Incident: blast radius becomes material
Incident --> Contained
Contained --> Recovering
Readiness is a capability state. A shell can remain available while a mutating, provider-backed, or export path is read-only, queued, or explicitly unavailable.
Operational inventory#
For each domain, maintain:
- supported UI/native/CLI entry points and deployment units;
- APIs, event topics, queues, scheduled cycles, workers, remote hosts, and callbacks;
- databases/schemas, blob stores, caches, search/vector/graph indices, audit and signing stores;
- provider/model/tool/DCC dependencies and credentials;
- feature flags, tenant scopes, configuration, secrets, limits, and ownership;
- dashboards, alerts, runbooks, recovery objectives, backups, restore targets, game-day scenarios, and release gates.
The generic estate inventory and domain inventories seed this register. Deployment discovery must reconcile them with what actually runs.
Health and readiness#
Liveness answers whether the process can make progress. Readiness answers whether it can safely accept its supported traffic. Readiness includes critical migrations, stores, queue/outbox, identity/policy configuration, required providers, and incompatible version fences. A noncritical provider may yield a declared degraded-ready state only when capability-specific degradation is safe and visible.
Workers expose lease/fencing health, backlog age/depth, attempts, poison/dead letters, throughput, latency, and result/reclamation state. DCC/remote hosts add host identity, supported versions/capabilities, project scope, heartbeat, current lease, artifact transfer, and command/result receipts.
Service-level indicators#
Measure user and control outcomes by domain and capability:
| Signal | Examples |
|---|---|
| Availability | Authorized project/source/session opens; command acceptance; evidence/history load; job status; export/download; remote-host reachability. |
| Latency | Interactive read/write, search, save acknowledgement, job queue-to-start, processing phase, provider/DCC round trip, projection lag, notification. |
| Correctness | Schema and invariant failures, conflict rate, duplicate effects, stale projections, result validation, lineage completeness, signature verification. |
| Safety and rights | Denials by reason, cross-tenant probes, quarantine, policy/rights blocks, expiry/deletion backlog, export exclusions, provider/model withdrawal. |
| Reliability | Retry success, poison jobs, lease loss, outbox lag, restore consistency, callback/update dedupe, cancellation/compensation, crash loops. |
| Experience | UI error/partial/offline state, save honesty, accessibility regressions, unsupported formats/versions, time to recover or complete a core journey. |
Metrics use bounded dimensions; tenant/resource ids and content do not become high-cardinality labels. Correlation and trace ids link sanitized logs, traces, audit, job, provider, and UI errors without exposing secrets.
Degradation#
Every dependency has a capability plan:
- what reads remain available and whether they are stale;
- what writes are blocked, queued, or safely local;
- whether jobs can start, continue, publish, retry, or cancel;
- what provider/model/tool fallback is compatible and authorized;
- what the user sees and what operators are alerted on;
- how the system detects recovery and reconciles queued/stale state;
- which data or results are withheld rather than guessed.
Fallback quality is verified for the same contract. Silently changing a model, renderer, codec, DCC version, or evaluation method can invalidate comparison and provenance even when a request succeeds.
Capacity and resource safety#
Plan limits for upload/ingest, media duration/resolution, archive expansion, projects/sources/anchors, concurrent sessions, job queue/backlog, GPU/CPU/ memory/disk, database connections/locks, blob bandwidth, search/index growth, provider quotas, exports, and remote hosts.
Admission control distinguishes user, tenant, capability, and global capacity. Backpressure is end to end; accepting work into an unbounded local queue is not protection. Processing sandboxes have time, memory, CPU, output, recursion, and network limits. Capacity failures yield explicit retry/limit guidance and do not corrupt source or history.
Backup, restore, and disaster recovery#
Recovery covers system-of-record relational data, blobs, revision/history, outbox/inbox and job state, audit/signing keys and evidence, plus the sources needed to rebuild search/vector/graph projections. External providers and DCC hosts require reconciliation because a database restore cannot rewind them.
A valid restore proves:
- tenant/resource counts and sampled semantic invariants;
- database-to-blob and source-to-derived reachability;
- history ancestry and no impossible job/lease state;
- outbox/inbox idempotency and external-effect reconciliation;
- projection rebuild and cutover consistency;
- rights, holds, expiry, deletion/tombstone, and revoked-access state;
- signature verification and key availability;
- application journeys, not only store health.
The
Yemaya backup/restore runbook
is the domain's worked example. Shared restore classes and reference edges are
implemented in libs/oshun/workbench-kit/src/restore-consistency.ts.
Rollout and rollback#
Roll out in explicit rings or cohorts with owner, entry criteria, measurement window, abort conditions, and promotion decision. Flags are tenant/capability scoped and observable. Schema, contract, worker, projection, client, and provider versions use a compatibility window; mixed-version behavior is tested.
Rollback is safe only if the previous version can interpret newly written records and external effects. Otherwise use forward-fix, disable new writes, keep compatible readers, or compensate. Never destroy migration or audit evidence to make the old build start.
Each release records source revision, generated artifacts, migrations, configuration/flags, dependency/model/tool versions, test and evaluation evidence, security/privacy review, rollout state, and approver.
Incident handling#
The common sequence is:
- Detect and declare the affected capability, tenants, data, and time window.
- Contain unsafe writes, providers, workers, routes, remote hosts, keys, or exports while retaining supported safe reads where possible.
- Preserve sanitized evidence and audit; do not copy sensitive content into an incident channel.
- Establish source of truth, blast radius, irreversible external effects, and deletion/rights implications.
- Repair, replay/rebuild/reconcile with idempotency and version fences.
- Verify the user journey, tenant isolation, data/provenance, and recovery objective before reopening.
- Communicate impact and residual risk; update runbooks, alerts, gates, and game-day coverage.
Domain-specific operational focus#
- Isis: GPU/provider capacity, model/workflow compatibility, queue and output lineage, safety/policy, consistency/parity, and generated asset reclamation.
- Metis: learning/assessment availability, evaluator/rubric versioning, progress/event ordering, credentials, integrations, and learner-data privacy.
- Yemaya: large media ingest/processing, parsers, analysis providers, projection/search/graph, rights expiry, evidence/export, deletion, and cross-domain adapters.
- Veritas: confidential sources, research/claim state, review/embargo, schedules, publish/delivery/correction/takedown, and social provider limits.
- Euterpe: realtime audio latency/glitches, collaboration consistency, native/browser/plugin compatibility, recording durability, and export.
- Aja: capture/reference ingest, motion jobs/models, skeleton/format compatibility, performer rights, large derived artifacts, and pipeline delivery.
- Bellona: bridge/host connectivity, engine/tool versions, remote command authorization, build/render queues, artifact transfer, approvals, and external-effect reconciliation.
Release evidence checklist#
- Targeted lint, type, unit, contract/schema, migration, and integration gates.
- Core browser/native/CLI journeys with success, denial, error, retry, partial, offline, conflict, cancellation, and recovery states.
- Accessibility and performance budgets for affected compositions.
- Security, tenant, rights, export, deletion, provider, and external-tool tests.
- Current dashboards/alerts/runbooks and a verified restore/rollback path when state or effects changed.
- Inventory and generated artifact freshness.
- Deployed smoke/canary and promotion/abort evidence.
Do not mark a capability complete because one package test passed. Completion means the supported entry point, real effect, enforcement, observability, failure handling, and recovery evidence agree.