Isis — Architecture

Technical architecture for the Generative AI Factory domain.

Isis is the generative AI content factory at the heart of the Oshun platform. Its job is to accept generation requests from any part of the platform — images, video, audio, 3D models, textures, Blender renders — and manage those requests from initial queuing through GPU execution to final output storage and provenance recording. Importantly, Isis knows nothing about why a given piece of content is being generated or how it will be used. It is a pure capability service: submit a job, get an output back, with a complete record of exactly how that output was produced.

The system is organized as three cooperating REST API services (job submission, workflow management, and output tracking), a GPU worker process, a CLI, and a web front-end. A shared Redis queue connects the API to the workers, and Redis Streams carry lifecycle events to the rest of the platform. A single PostgreSQL database (via Prisma ORM) stores all persistent state.

System Overview#

Isis provides:

• Job submission via REST API (Hono, default port 3000)
• GPU worker processing with ComfyUI, Blender, and custom executors
• Workflow management with versioning and templates (port 3001)
• Output tracking with full provenance and lineage graph (port 3002)
• Model registry for checkpoint and LoRA management
• Client SDK (@isis/client) and CLI for programmatic access

Isis acts as the creative engine of the platform — other domains submit generation requests, and Isis manages the complete lifecycle from job queuing through GPU execution to output storage and provenance recording.

High-Level Topology#

The following diagram shows how the five major runtime components relate to one another. The CLI and other domain clients talk to the three REST services; the generation API enqueues work to the GPU worker via Redis; and all services share a single PostgreSQL database and a MinIO/S3 object store.

                     +------------------+
                     |    isis CLI      |
                     |   (Commander)    |
                     +--------+---------+
                              |
             +----------------+----------------+
             |                |                 |
    +--------v--------+ +----v----------+ +----v-----------+
    | generation-api  | | workflow-     | | output-        |
    | Port 3000       | | registry      | | registry       |
    | (Hono)          | | Port 3001     | | Port 3002      |
    | /api/v1/jobs    | | (Hono)        | | (Hono)         |
    | /api/v1/models  | | /api/v1/      | | /api/v1/       |
    +--------+--------+ | workflows     | | outputs        |
             |           | /api/v1/      | | /api/v1/       |
             |           | templates     | | policies       |
             |           +---------------+ +----------------+
             |
    +--------v--------+
    |   gpu-worker    |
    |   Redis queue   |
    |   GPU executors |
    |   - texture     |
    |   - blender     |
    |   - gaussian    |
    |   - mesh        |
    +-----------------+
             |
    +--------v--------+
    |   PostgreSQL    |   (ISIS_DATABASE_URL)
    |   (Prisma ORM)  |
    +-----------------+
             |
    +--------v--------+
    |    MinIO / S3   |   (generated outputs)
    +-----------------+
             |
    +--------v--------+
    |   Redis         |   (job queue + @oshun/event-bus)
    +-----------------+

Service Topology#

Applications (6)#

Isis deploys as six distinct applications. Three are REST APIs that accept external requests; one is a background worker that processes GPU jobs; one is a CLI for scripted workflows; and one is a Next.js web front-end. REST services read their listen port from process.env.PORT, defaulting as shown below (apps/isis/*/src/index.ts).

Application Layer#

generation-api (default Port 3000)#

The generation API is the single entry point for all job submission. It authenticates requests, validates inputs with Zod, writes a GenerationJob record to the database, and enqueues the job for a GPU worker to pick up. It also exposes model management, workflow execution triggers, webhook subscriptions, and provider callbacks.

Middleware stack (Hono):

• CORS with configurable origins
• Secure headers (CSP, HSTS, X-Frame-Options, etc.)
• Pretty JSON for development
• Rate limiting: 100 requests per 60 seconds on /api/*
• JWT authentication middleware (fail-fast: requires JWT_SECRET in production)
• Zod request/response validation via @hono/zod-validator

Core services:

• JobService — submit, list, cancel, get stats, queue health
• ModelService — register, list, get, update, generate upload URLs
• ComfyUIService — workflow execution via ComfyUI WebSocket protocol

The app mounts five route groups: jobs, models, workflows, webhook-subscriptions, plus /webhooks (provider callbacks). The workflow routes validate a typed contract for each of 32 asset-pack workflow recipes.

Health endpoints:

• GET /health — Simple liveness probe (always 200 if process alive)
• GET /ready — Readiness probe: runs job-service, model-service, and ComfyUI-service health checks in parallel and returns 200/503 with a per-dependency breakdown (jobs DB, models DB, queue, storage, provider)

gpu-worker#

The GPU worker is a long-running Node.js process — it does not expose any HTTP port. Instead it polls a Redis queue continuously, deserializes incoming JobEnvelope messages, and routes each job to the appropriate executor. When a job finishes, the worker uploads the output to S3, writes GeneratedOutput and Provenance database records, and publishes a completion event on the event bus.

The flow through the worker on every job is:

  Poll Redis queue (ISIS_QUEUE_NAME)
         │
         ▼
  Deserialize JobEnvelope
         │
  Route by GenerationType:
         ├── TEXTURE_UPSCALE  → texture-upscale-executor
         ├── BLENDER_RENDER   → blender-render-executor
         ├── GAUSSIAN_SPLATTING → gaussian-splatting-executor
         ├── MESH_PROCESSING  → mesh-processing-executor
         └── All other types  → generic executor (provider-routed)
         │
         ▼
  Execute (may call AI provider API)
         │
         ▼
  Upload output to S3
         │
         ▼
  Create GeneratedOutput + Provenance records
         │
         ▼
  Update GenerationJob status → COMPLETED | FAILED
         │
         ▼
  Publish isis.job.completed event

Configuration:

• GPU_DEVICES — comma-separated CUDA device indices
• WORKER_TYPE — one of seven worker types (texture-upscale, mesh-processing, blender-render, gaussian-splatting, video-generation, neural-bake, general)
• QUEUE_NAME — Redis queue to poll
• WORKER_ID — unique worker identifier registered in GpuWorker table

Lifecycle events emitted (internal): worker:started, worker:stopped, job:started, job:progress, job:completed, job:failed

Graceful shutdown: On SIGINT/SIGTERM, the worker finishes the current job before exiting.

workflow-registry (Port 3001)#

The workflow registry stores and versions ComfyUI workflow definitions. It is separate from the generation API so that workflow authoring tools and administrative interfaces can talk directly to it without going through the job submission path. It has two storage backends — postgres (production) and in-memory (test/dev override) — and exposes full-text search via PostgreSQL tsvector when database-backed.

Key capabilities:

• Full CRUD with semver versioning, changelog tracking, and star/favorite system
• Visibility controls: PRIVATE, TEAM, ORGANIZATION, PUBLIC
• Staging-recipe routes for promotion workflows

output-registry (Port 3002)#

The output registry is the long-term home for every file that Isis generates. Rather than baking lifecycle management into the generation API, it is separated here so that storage policy decisions — tiering, retention, expiry — can evolve independently of job submission.

• Tracks every output file with hash, dimensions, storage tier, and access statistics
• Storage tiering — Automatic lifecycle management: HOT → WARM → COLD → GLACIER based on RetentionPolicy rules and access patterns
• Lineage graph — Directed graph of LineageEdge records tracking derivation between outputs (e.g., upscaled-from, converted-from)
• Provenance — Immutable Provenance records capturing full generation context for reproducibility
• File deduplication — Content hash uniqueness prevents duplicate storage

cli#

Commander.js CLI (isis) for scripted generation workflows and CI/CD integration. Authenticates via JWT token or API key stored in ~/.isis/config.yaml. The CLI covers seven command groups: generate, jobs, workflows, outputs, config, health, and auth.

Library Architecture#

The 55 libraries in libs/isis/ are organized by concern. Understanding which group a library belongs to tells you what kind of dependency it is: core platform libraries are stable and depended on widely; provider adapters change when AI vendor APIs change; 3D generation libraries form a largely self-contained sub-system. What follows groups the 55 directories (54 TypeScript @isis/* packages plus one Python package) by that concern.

Core platform (6)#

These six libraries form the foundation that every other Isis library or application may depend on.

AI providers, LLM, and orchestration (10)#

These libraries abstract every external AI vendor behind a common interface, and provide higher-level orchestration (batching, consensus, ReAct agents) on top.

ComfyUI and workflow authoring (5)#

ComfyUI is the primary visual workflow engine. These libraries cover everything from low-level WebSocket communication with a running ComfyUI instance up to higher-level workflow-class catalogs and custom Python nodes.

Image, video, and audio (11)#

This group covers the full media-generation surface area beyond simple text-to-image: AI video generation and enhancement, geometry extraction from video, all audio and voice synthesis capabilities, and AI texturing for 3D assets.

3D generation (16)#

The 3D generation sub-system is the most complex part of Isis, covering multi-provider cloud generation, quality gating, rigging, post-processing, scene assembly, and governance. These 16 libraries form a largely self-contained 3D pipeline.

Models, quality, and operations (7)#

The final group covers everything that happens around the edges of generation: fine-tuning models to produce new capabilities, detecting misuse, gating access by entitlement, surfacing outputs in a gallery, and managing the RunPod serverless GPU endpoint.

(The apps/isis/web front-end is packaged as @isis/web.)

Data Flow#

Understanding how data moves through Isis is key to debugging and extending it. The three flows below cover the common case (single generation job), the workflow-driven case, and the output lineage tracking that links derived assets back to their origins.

Generation Pipeline#

This is the most common path: a client submits a job, the API queues it, a GPU worker executes it, and the resulting file lands in object storage with its provenance recorded.

   Client / CLI / Other Domain      Isis Services              Storage
   ─────────────────────────   ──────────────────────   ─────────────────

   POST /api/v1/jobs            generation-api
   {                                │
     type: TEXT_TO_IMAGE,           │ validate JWT
     prompt: "...",                 │ validate Zod schema
     model: "sdxl-1.0"            │ create GenerationJob (PENDING)
   }                                │ enqueue to Redis
                                    │
   ← 202 Accepted                   │
   { id: "job_xxx", status: PENDING }│
                               gpu-worker
                                    │ poll queue
                                    │
                                    ├─ resolve ModelRegistry
                                    │  by modelId
                                    │
                                    ├─ execute via
                                    │  ComfyUI / Executor
                                    │  (or AI provider API)
                                    │
                                    ├─ emit job:progress events
                                    │  (progress 0→100%)
                                    │
                                    ├─ upload output to S3 ──────────→ MinIO
                                    │
                                    ├─ create GeneratedOutput
                                    │  {hash, dimensions, tier: HOT}
                                    │
                                    ├─ create Provenance
                                    │  {model, params, seed, gpu, cost}
                                    │
                                    ├─ update job → COMPLETED
                                    │
                                    └─ publish isis.asset.generated ──→ event bus
                               output-registry
                                    │ store manifest
                                    │ evaluate retention policy
                                    │ queue tier transitions

Workflow Execution Flow#

When a job is submitted with a workflowId, the generation API loads the workflow definition from the workflow registry, validates the job parameters against the workflow's schema, and enqueues the job with the full workflow context attached. The GPU worker then drives ComfyUI over WebSocket to execute the graph.

   User                    workflow-registry       generation-api
   ────                    ─────────────────       ──────────────

   Create workflow
   with nodes/edges  ──→  Store definition
                          + version (1.0.0)
                               │
   Submit job with             │
   workflowId ─────────────────────────────────→  Load workflow
                                                   definition from DB
                                                   │
                                                   Validate params
                                                   against schema
                                                   │
                                                   Enqueue with
                                                   workflow context
                                                   │
                                             gpu-worker
                                                   │
                                             Execute via
                                             ComfyUI WebSocket
                                                   │
                                             Store output
                                             + provenance

Output Lineage Flow#

Every derived output — an upscaled image, a converted mesh, an engine-native asset — is linked back to its origin via LineageEdge records. This makes it possible to trace any asset backwards to its original generation job and parameters, and to reconstruct the full derivation chain for compliance or reproducibility purposes.

   Original Image               Upscaled              Engine Asset
   ─────────────                ────────              ────────────

   GeneratedOutput  ─UPSCALED_FROM─→  GeneratedOutput
   (job: t2i)                         (job: upscale)
   (512×512)                          (2048×2048)
                                            │
                                      CONVERTED_FROM
                                            │
                                            ▼
                                      GeneratedOutput
                                      (.uasset format)
                                      bellonaAssetId set
                                      in Yemaya Asset record

Cross-Domain Integration#

Isis is a pure capability domain — it provides generation services to other domains and does not depend on any other domain. This boundary is deliberate: if Isis depended on Yemaya or Bellona, a change to those domains could break generation for everyone. Instead, Isis speaks only in opaque IDs (projectId, userId) that originating domains provide and receive back.

Event Bus Architecture#

Isis uses @isis/event-publisher (wrapping @oshun/event-bus) to publish lifecycle events on Redis Streams under the oshun:events key prefix. Any domain interested in generation outcomes subscribes to these events rather than polling the Isis API:

Isis generation-api / gpu-worker
    │
    └── @isis/event-publisher
            │
            └── @oshun/event-bus (Redis Streams, keyPrefix "oshun:events")
                    │
                    ├── isis.job.* events  → job lifecycle (queued/started/
                    │                         progress/completed/failed/cancelled)
                    ├── isis.asset.generated → new asset available
                    ├── isis.workflow.*    → workflow registered/updated
                    └── isis.model.loaded  → model loaded onto a GPU worker

Event types and payload schemas are defined canonically in @oshun/contracts (IsisEventTypes). Event publishing is best-effort: a publish failure is logged and swallowed so it never breaks the generation flow.

Inbound Integration#

Other domains (Yemaya, Hathor, Bellona, Lilith) submit generation jobs directly to the Isis REST API or via the @isis/client SDK. Jobs carry a projectId cross-reference that is stored verbatim on the GenerationJob record. Isis never resolves what that project represents — that is always the caller's responsibility. The echoed projectId in completion events is how the originating domain correlates results back to its own records:

Yemaya publish yemaya.generation.requested
  → Isis handler creates GenerationJob{projectId: "prj_xyz"}
  → Job executes, output stored
  → Isis publishes isis.job.completed{projectId: "prj_xyz"}
  → Yemaya handler resolves Project by projectId, creates Asset

Canary Rollout and Operational Tooling#

Isis includes production-grade operational tooling for safe rollouts and incident management, documented in docs/domains/isis/remediation/ and docs/domains/isis/remediation-v2/.

Canary Queue Rollout#

Rather than doing a hard cutover when the queue producer changes, Isis supports a gradual canary rollout controlled by environment variables. Traffic is split between the old and new queue at a configured percentage, and SLO gates determine whether to promote or roll back:

ISIS_QUEUE_PRODUCER_ENABLED=true          # Master switch
ISIS_QUEUE_PRODUCER_CANARY_ENABLED=false  # Canary mode
ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=10  # 10% of traffic to new queue
ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS=   # Allowlist for canary routing

Canary advancement is automated via scripts/isis/advance_queue_canary_rollout.sh with SLO-gated promotion/rollback decisions.

Operational Scripts#

A set of shell scripts in scripts/isis/ supports the full release and validation lifecycle:

SLO Definitions#

The following SLOs are defined in docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.json and gate canary promotion decisions:

• Job submission success rate: >99.5%
• Job completion latency (p99): type-dependent (image < 30s, 3D < 300s)
• Output storage durability: >99.999%
• API availability: >99.9%

Deployment Architecture#

Container Strategy#

Each Isis service deploys as its own container image. The API services are stateless and scale horizontally behind a load balancer; the GPU worker scales by queue depth and is tied to GPU node pools.

GPU Infrastructure#

GPU workers are deployed on GPU-enabled nodes. The GPU_DEVICES environment variable specifies which CUDA devices the worker can use. Multiple worker instances can run on the same node (one per GPU) or across multiple nodes.

RunPod serverless is used for on-demand GPU capacity spikes without maintaining persistent GPU instances (per ADR-0003).

Development Stack#

To bring up Isis locally with GPU support:

# Start Isis local stack with GPU profile
docker compose -f docker/docker-compose.yml --profile gpu up -d \
  postgres redis qdrant isis-api isis-generator

Technology Stack#

The following table records the technology choice for each layer and the rationale behind it. Cross-referencing this with the ADR list (in the specifications doc) gives the full decision history.

Design Principles#

The following principles explain why the architecture is shaped the way it is, not just what it does. Each one reflects a deliberate trade-off made during the system's design.

1. Generation-as-a-service — Isis is a pure capability service with no domain knowledge about what the content will be used for. Job context (projectId, userId) flows through as opaque references. This isolation means changes to Yemaya or Bellona's data models never require changes to Isis.

2. Full provenance for every output — Every generated file has an immutable Provenance record capturing model, parameters, seed, and cost. This enables reproducibility, cost attribution, and compliance auditing. You can always answer "exactly how was this file produced?"

3. Lineage over mutation — Outputs are never modified in place. Transformations (upscaling, conversion) create new GeneratedOutput records linked by LineageEdge records. The lineage graph is append-only, which means the history of every asset is complete and unambiguous.

4. Provider abstraction — @isis/ai-providers abstracts all AI provider differences behind a uniform interface. Generation type determines which providers are eligible; the system routes to the best available provider. Adding a new provider requires only a new adapter, not changes to job submission logic.

5. Poll-based queue for reliability — The Redis poll model (ADR-0001) was chosen over pub/sub push because it provides better at-least-once delivery guarantees and makes worker restarts safe without job loss.

6. Canary-first deployments — All queue and API changes go through canary rollout with SLO-gated promotion. Rollback criteria are pre-defined and automated.