Oshun Platform · Guides & deep dives

Workbench Operations, Observability, and Release

For each domain, maintain:

10sections5 minread1table

On this page

Workbench operations cover interactive shells, APIs, workers, queues, databases, object storage, search/graph projections, media/model providers, DCC/engine bridges, and external delivery. A green route alone is not workbench health; the supported journey and its recovery path must be observable end to end.

State diagramState diagram: 9 states and 14 transitions.dependency or SLO breachwrites cannot be proven safedurable bounded backlog is safeno honest capability remainsstores and projections reconcileworkers providers and fences recoverconfiguration or dependency restoredjourneys receipts and invariants passvalidation or capacity failsisolation rights safety or external-effect breachblast radius becomes materialReadyDegradedReadOnlyQueuingUnavailableRecoveringIncidentContained
State machine: State diagram. State diagram: 9 states and 14 transitions.

This picture is wider than it can be shown here. Move across it, or read it as text.

Drawn from its Mermaid source.

As text
Parts:
- Start (initial)
- Ready
- Degraded
- ReadOnly
- Queuing
- Unavailable
- Recovering
- Incident
- Contained

Connections:
- Start to Ready
- Ready to Degraded: dependency or SLO breach
- Degraded to ReadOnly: writes cannot be proven safe
- Degraded to Queuing: durable bounded backlog is safe
- Degraded to Unavailable: no honest capability remains
- ReadOnly to Recovering: stores and projections reconcile
- Queuing to Recovering: workers providers and fences recover
- Unavailable to Recovering: configuration or dependency restored
- Recovering to Ready: journeys receipts and invariants pass
- Recovering to Degraded: validation or capacity fails
- Ready to Incident: isolation rights safety or external-effect breach
- Degraded to Incident: blast radius becomes material
- Incident to Contained
- Contained to Recovering
Show source
stateDiagram-v2
  [*] --> Ready
  Ready --> Degraded: dependency or SLO breach
  Degraded --> ReadOnly: writes cannot be proven safe
  Degraded --> Queuing: durable bounded backlog is safe
  Degraded --> Unavailable: no honest capability remains
  ReadOnly --> Recovering: stores and projections reconcile
  Queuing --> Recovering: workers providers and fences recover
  Unavailable --> Recovering: configuration or dependency restored
  Recovering --> Ready: journeys receipts and invariants pass
  Recovering --> Degraded: validation or capacity fails
  Ready --> Incident: isolation rights safety or external-effect breach
  Degraded --> Incident: blast radius becomes material
  Incident --> Contained
  Contained --> Recovering

Readiness is a capability state. A shell can remain available while a mutating, provider-backed, or export path is read-only, queued, or explicitly unavailable.

Operational inventory#

For each domain, maintain:

  • supported UI/native/CLI entry points and deployment units;
  • APIs, event topics, queues, scheduled cycles, workers, remote hosts, and callbacks;
  • databases/schemas, blob stores, caches, search/vector/graph indices, audit and signing stores;
  • provider/model/tool/DCC dependencies and credentials;
  • feature flags, tenant scopes, configuration, secrets, limits, and ownership;
  • dashboards, alerts, runbooks, recovery objectives, backups, restore targets, game-day scenarios, and release gates.

The generic estate inventory and domain inventories seed this register. Deployment discovery must reconcile them with what actually runs.

Health and readiness#

Liveness answers whether the process can make progress. Readiness answers whether it can safely accept its supported traffic. Readiness includes critical migrations, stores, queue/outbox, identity/policy configuration, required providers, and incompatible version fences. A noncritical provider may yield a declared degraded-ready state only when capability-specific degradation is safe and visible.

Workers expose lease/fencing health, backlog age/depth, attempts, poison/dead letters, throughput, latency, and result/reclamation state. DCC/remote hosts add host identity, supported versions/capabilities, project scope, heartbeat, current lease, artifact transfer, and command/result receipts.

Service-level indicators#

Measure user and control outcomes by domain and capability:

Signal Examples
Availability Authorized project/source/session opens; command acceptance; evidence/history load; job status; export/download; remote-host reachability.
Latency Interactive read/write, search, save acknowledgement, job queue-to-start, processing phase, provider/DCC round trip, projection lag, notification.
Correctness Schema and invariant failures, conflict rate, duplicate effects, stale projections, result validation, lineage completeness, signature verification.
Safety and rights Denials by reason, cross-tenant probes, quarantine, policy/rights blocks, expiry/deletion backlog, export exclusions, provider/model withdrawal.
Reliability Retry success, poison jobs, lease loss, outbox lag, restore consistency, callback/update dedupe, cancellation/compensation, crash loops.
Experience UI error/partial/offline state, save honesty, accessibility regressions, unsupported formats/versions, time to recover or complete a core journey.

Metrics use bounded dimensions; tenant/resource ids and content do not become high-cardinality labels. Correlation and trace ids link sanitized logs, traces, audit, job, provider, and UI errors without exposing secrets.

Degradation#

Every dependency has a capability plan:

  • what reads remain available and whether they are stale;
  • what writes are blocked, queued, or safely local;
  • whether jobs can start, continue, publish, retry, or cancel;
  • what provider/model/tool fallback is compatible and authorized;
  • what the user sees and what operators are alerted on;
  • how the system detects recovery and reconciles queued/stale state;
  • which data or results are withheld rather than guessed.

Fallback quality is verified for the same contract. Silently changing a model, renderer, codec, DCC version, or evaluation method can invalidate comparison and provenance even when a request succeeds.

Capacity and resource safety#

Plan limits for upload/ingest, media duration/resolution, archive expansion, projects/sources/anchors, concurrent sessions, job queue/backlog, GPU/CPU/ memory/disk, database connections/locks, blob bandwidth, search/index growth, provider quotas, exports, and remote hosts.

Admission control distinguishes user, tenant, capability, and global capacity. Backpressure is end to end; accepting work into an unbounded local queue is not protection. Processing sandboxes have time, memory, CPU, output, recursion, and network limits. Capacity failures yield explicit retry/limit guidance and do not corrupt source or history.

Backup, restore, and disaster recovery#

Recovery covers system-of-record relational data, blobs, revision/history, outbox/inbox and job state, audit/signing keys and evidence, plus the sources needed to rebuild search/vector/graph projections. External providers and DCC hosts require reconciliation because a database restore cannot rewind them.

A valid restore proves:

  • tenant/resource counts and sampled semantic invariants;
  • database-to-blob and source-to-derived reachability;
  • history ancestry and no impossible job/lease state;
  • outbox/inbox idempotency and external-effect reconciliation;
  • projection rebuild and cutover consistency;
  • rights, holds, expiry, deletion/tombstone, and revoked-access state;
  • signature verification and key availability;
  • application journeys, not only store health.

The Yemaya backup/restore runbook is the domain's worked example. Shared restore classes and reference edges are implemented in libs/oshun/workbench-kit/src/restore-consistency.ts.

Rollout and rollback#

Roll out in explicit rings or cohorts with owner, entry criteria, measurement window, abort conditions, and promotion decision. Flags are tenant/capability scoped and observable. Schema, contract, worker, projection, client, and provider versions use a compatibility window; mixed-version behavior is tested.

Rollback is safe only if the previous version can interpret newly written records and external effects. Otherwise use forward-fix, disable new writes, keep compatible readers, or compensate. Never destroy migration or audit evidence to make the old build start.

Each release records source revision, generated artifacts, migrations, configuration/flags, dependency/model/tool versions, test and evaluation evidence, security/privacy review, rollout state, and approver.

Incident handling#

The common sequence is:

  1. Detect and declare the affected capability, tenants, data, and time window.
  2. Contain unsafe writes, providers, workers, routes, remote hosts, keys, or exports while retaining supported safe reads where possible.
  3. Preserve sanitized evidence and audit; do not copy sensitive content into an incident channel.
  4. Establish source of truth, blast radius, irreversible external effects, and deletion/rights implications.
  5. Repair, replay/rebuild/reconcile with idempotency and version fences.
  6. Verify the user journey, tenant isolation, data/provenance, and recovery objective before reopening.
  7. Communicate impact and residual risk; update runbooks, alerts, gates, and game-day coverage.

Domain-specific operational focus#

  • Isis: GPU/provider capacity, model/workflow compatibility, queue and output lineage, safety/policy, consistency/parity, and generated asset reclamation.
  • Metis: learning/assessment availability, evaluator/rubric versioning, progress/event ordering, credentials, integrations, and learner-data privacy.
  • Yemaya: large media ingest/processing, parsers, analysis providers, projection/search/graph, rights expiry, evidence/export, deletion, and cross-domain adapters.
  • Veritas: confidential sources, research/claim state, review/embargo, schedules, publish/delivery/correction/takedown, and social provider limits.
  • Euterpe: realtime audio latency/glitches, collaboration consistency, native/browser/plugin compatibility, recording durability, and export.
  • Aja: capture/reference ingest, motion jobs/models, skeleton/format compatibility, performer rights, large derived artifacts, and pipeline delivery.
  • Bellona: bridge/host connectivity, engine/tool versions, remote command authorization, build/render queues, artifact transfer, approvals, and external-effect reconciliation.

Release evidence checklist#

  • Targeted lint, type, unit, contract/schema, migration, and integration gates.
  • Core browser/native/CLI journeys with success, denial, error, retry, partial, offline, conflict, cancellation, and recovery states.
  • Accessibility and performance budgets for affected compositions.
  • Security, tenant, rights, export, deletion, provider, and external-tool tests.
  • Current dashboards/alerts/runbooks and a verified restore/rollback path when state or effects changed.
  • Inventory and generated artifact freshness.
  • Deployed smoke/canary and promotion/abort evidence.

Do not mark a capability complete because one package test passed. Completion means the supported entry point, real effect, enforcement, observability, failure handling, and recovery evidence agree.