Eve library

Isis Domain - Developer Guide

Guide

Revision 1

On this page

Isis is the Generative AI Factory domain. It provides GPU-accelerated generation services for images, video, 3D models, audio, and more. It manages ComfyUI workflows, GPU worker orchestration, model registries, output provenance tracking, and multi-format generation pipelines.


Prerequisites#

Required Software#

Software Minimum Version Purpose
Node.js >= 18.0.0 Runtime for API services
pnpm >= 9.0.0 Package manager (monorepo workspace)
Docker >= 24.0 Infrastructure and GPU worker images
TypeScript ^6.0.3 Type checking and compilation

For GPU Worker Development#

Software Version Purpose
NVIDIA Driver >= 535 GPU access for workers
CUDA Toolkit >= 12.0 GPU compute for AI models
nvidia-docker >= 2.0 Docker GPU passthrough
Blender >= 4.0 Blender render executor

GPU hardware is only required for running gpu-worker executors. All other services (generation-api, output-registry, workflow-registry, cli) run on standard hardware.


Project Structure#

text
apps/isis/
  generation-api/     # Main API - job submission, status tracking, model management
  gpu-worker/         # GPU job processor (BullMQ worker with executor plugins)
  output-registry/    # Output file management API (storage, retrieval, retention)
  workflow-registry/  # Workflow template management API
  cli/                # Command-line interface for generation, jobs, workflows

libs/isis/
  client/             # TypeScript API client (@isis/client)
  database/           # Prisma schema and generated client
  audio-generation/   # Audio/voice/music generation (ElevenLabs, Suno, Udio)
  workflows/          # Workflow JSON definitions (ComfyUI, Blender, etc.)
  event-publisher/    # Cross-domain event publisher (@isis/event-publisher)

Workflow Library Categories#

The libs/isis/workflows/ directory contains pre-built workflow definitions:

  • sacred-art/ - Mandala, thangka, yantra, celtic knotwork generators
  • fractal/ - Mandelbrot, Julia, flame fractal, Lorenz attractor
  • avatar/ - Face swap, cosmetic layers, consistent character
  • utility/ - Background removal, style transfer, batch operations
  • nft/ - Rarity enhancement workflows

Local Development Setup#

1. Prerequisites#

bash
# Required
docker --version
docker compose version
pnpm --version

GPU acceleration is optional for local startup, but if you want to run the worker against local GPUs you need NVIDIA drivers and NVIDIA Container Toolkit installed on the host.

2. Start Isis Services with Compose (GPU Profile)#

bash
# From monorepo root (full Isis profile including GPU worker)
docker compose -f docker/docker-compose.yml --profile gpu up -d \
  postgres redis qdrant isis-api isis-generator

If you do not have local GPU support, start only the API and its dependencies:

bash
docker compose -f docker/docker-compose.yml --profile gpu up -d \
  postgres redis qdrant isis-api

Default API host port in this compose stack is http://localhost:3003.

3. Validate Compose Wiring and Smoke Health#

bash
# Validate APP_PATH references used by Isis compose services
bash scripts/isis/validate_compose_app_paths.sh

# Compose smoke test (config validation only; fast)
CONFIG_ONLY=true scripts/isis/smoke_compose_health.sh

# Full smoke test: boots Isis compose services and checks /health + /ready
# Note: first run can be slow because Docker images may need to build/pull.
scripts/isis/smoke_compose_health.sh

4. Verify Health Endpoints Manually#

bash
curl -fsS http://localhost:${ISIS_API_PORT:-3003}/health

# /ready may return 200 (ready) or 503 (degraded/unready) depending on optional dependencies.
curl -s -o /tmp/isis-ready.json -w "%{http_code}\n" \
  http://localhost:${ISIS_API_PORT:-3003}/ready

5. Environment Variables (Local Reference)#

Variable Default Used By
ISIS_DATABASE_URL (from root .env) database lib
REDIS_URL redis://localhost:6379 generation-api, gpu-worker
ISIS_QUEUE_NAME isis:jobs:submit generation-api
ISIS_QUEUE_REDIS_URL (optional in dev) generation-api
ISIS_QUEUE_PRODUCER_ENABLED true generation-api
ISIS_QUEUE_PRODUCER_CANARY_ENABLED false generation-api
ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE 0 generation-api
ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS (optional CSV) generation-api
REDIS_HOST localhost gpu-worker
REDIS_PORT 6379 gpu-worker
PORT 3000 generation-api
HOST localhost generation-api
ISIS_MODEL_MALWARE_SCAN_ENABLED true generation-api
ISIS_MODEL_ABANDONED_UPLOAD_RETENTION_HOURS 24 generation-api

For production-like startup, generation-api requires queue configuration (ISIS_QUEUE_NAME and either ISIS_QUEUE_REDIS_URL or REDIS_URL) when queue dispatch is active:

  • ISIS_QUEUE_PRODUCER_ENABLED=true, or
  • ISIS_QUEUE_PRODUCER_ENABLED=false with ISIS_QUEUE_PRODUCER_CANARY_ENABLED=true and ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE>0.

6. Verify Canary SLO and Cost Gate (Release Readiness)#

bash
ISIS_DATABASE_URL=postgres://... \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
scripts/isis/verify_canary_slo_cost.sh

To enforce gate failure in automation:

bash
ISIS_DATABASE_URL=postgres://... \
FAIL_ON_BREACH=true \
scripts/isis/verify_canary_slo_cost.sh

Guide and threshold details: docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md

7. Advance Canary Ramp with Auto-Rollback Decisioning#

bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh

Auto-rollback (decision + non-zero exit) on threshold failure:

bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=true \
scripts/isis/advance_queue_canary_rollout.sh

Guide and callback integration details: docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md

8. Run Release Game-Day Harness (Pre-Cutover)#

bash
ISIS_DATABASE_URL=postgres://... \
ISIS_API_BASE_URL=https://<isis-api-host> \
CURRENT_PERCENTAGE=10 \
STRICT=true \
scripts/isis/run_release_gameday.sh

Plan and reporting templates:

  • docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md
  • docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md

9. Run Post-Cutover Validation and Sign-Off Harness#

bash
ISIS_API_BASE_URL=https://<isis-api-host> \
ISIS_DATABASE_URL=postgres://... \
STRICT=true \
scripts/isis/run_post_cutover_validation.sh

Checklist and sign-off templates:

  • docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md
  • docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md

10. Install Dependencies (Direct Local Runs)#

bash
pnpm install

11. Seed the Database (optional for direct runs)#

bash
cd libs/isis/database
npx tsx prisma/seed.ts       # Production-like seed data
npx tsx prisma/seed-test.ts  # Test fixtures

Running Services#

Generation API#

The main entry point for submitting generation jobs, querying status, and managing models. Built with Hono and Zod OpenAPI.

bash
# Development (hot reload)
cd apps/isis/generation-api
pnpm dev   # tsx watch src/index.ts

# Via Nx
pnpm nx serve @isis/generation-api

The API exposes:

  • POST /jobs - Submit generation jobs
  • GET /jobs/:id - Check job status
  • GET /models - List available models
  • POST /models - Register new models

GPU Worker#

The GPU worker pulls jobs from BullMQ queues and routes them to the appropriate executor. Each executor handles a specific generation type.

bash
# Development (no GPU needed for the service itself)
pnpm nx serve @isis/gpu-worker

# Specific configuration
pnpm nx serve @isis/gpu-worker --configuration=development

Available executors:

Executor File Purpose
Blender Render blender-render-executor.ts Blender scene rendering
Mesh Processing mesh-processing-executor.ts 3D mesh operations
Texture Upscale texture-upscale-executor.ts AI texture upscaling

Output Registry#

Manages generated output files with storage lifecycle, retention policies, and provenance tracking.

bash
pnpm nx serve @isis/output-registry

Workflow Registry#

Manages workflow templates and versioning.

bash
pnpm nx serve @isis/workflow-registry

CLI#

bash
# Development
pnpm nx dev @isis/cli

# Available commands
isis generate   # Submit a generation job
isis jobs       # List/manage jobs
isis outputs    # Browse generated outputs
isis workflows  # Manage workflows
isis auth       # Authentication
isis health     # Service health check
isis config     # Configuration management
isis jobs triage # Operator stuck-job and DLQ triage
isis jobs replay # Replay dead-letter outbox entries

Building#

Build All Isis Projects#

bash
pnpm nx run-many --target=build --projects=tag:scope:isis

Build Individual Projects#

bash
# Generation API (tsc)
pnpm nx build @isis/generation-api

# GPU Worker (esbuild, bundled)
pnpm nx build @isis/gpu-worker
pnpm nx build @isis/gpu-worker --configuration=production  # Minified

# Client library
pnpm nx build @isis/client

Build GPU Worker Docker Images#

The gpu-worker has specialized Docker images for each executor type:

bash
pnpm nx docker-build @isis/gpu-worker

This builds four images:

  • isis/gpu-worker:base - Base image with shared dependencies
  • isis/gpu-worker:texture-upscale - Texture upscaling executor
  • isis/gpu-worker:mesh-processing - Mesh processing executor
  • isis/gpu-worker:blender-render - Blender rendering executor

Dockerfiles are located in apps/isis/gpu-worker/src/docker/.


Testing#

Test Runner Configuration#

All Isis apps use Vitest with the @nx/vite:test executor.

Vitest configuration (vitest.config.ts):

typescript
export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    include: ['src/**/*.{test,spec}.ts'],
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
      exclude: ['node_modules/', 'dist/', '**/*.d.ts'],
    },
  },
});

Running Tests#

bash
# All Isis projects
pnpm nx run-many --target=test --projects=tag:scope:isis

# Individual projects
pnpm nx test @isis/generation-api
pnpm nx test @isis/gpu-worker
pnpm nx test @isis/output-registry
pnpm nx test @isis/workflow-registry
pnpm nx test @isis/client

Testing Without a GPU#

The GPU worker's executors can be tested without actual GPU hardware:

  1. Unit tests - All executors extend BaseExecutor which defines a standard interface. Mock the executor in tests:

    typescript
    import { vi } from 'vitest';
    
    const mockExecutor = {
      initialize: vi.fn().mockResolvedValue(undefined),
      execute: vi.fn().mockResolvedValue({ status: 'completed', outputs: [] }),
      shutdown: vi.fn().mockResolvedValue(undefined),
    };
    
  2. Integration tests - The GPUWorker class can be instantiated with gpuMonitoring: false in the config. GPU info calls return placeholder data when monitoring is disabled.

  3. Mock worker approach - Set the worker config's gpuDevices to [0] and disable cleanup. The worker will process jobs through the executor pipeline without requiring real GPU hardware.

Linting#

bash
pnpm nx run-many --target=lint --projects=tag:scope:isis
pnpm nx lint @isis/generation-api

Database Schema#

The Isis database (libs/isis/database/prisma/schema.prisma) uses PostgreSQL with the following key models:

Model Purpose
GenerationJob Job records with type, status, prompt, model, worker assignment
Workflow Workflow definitions with versioning, parameters, engine type
WorkflowVersion Workflow version history with definition snapshots
WorkflowTemplate Pre-built workflow templates
WorkflowStar User favorites on workflows
GeneratedOutput Output files with storage location, dimensions, lineage
LineageEdge Derivation relationships between outputs
Provenance Full provenance records (model, LoRAs, seed, GPU, cost)
ModelRegistry AI model/checkpoint registry
RetentionPolicy Storage lifecycle rules
GpuWorker GPU worker registration and health tracking
AuditLog Change tracking

Key Enums#

  • GenerationType: TEXT_TO_IMAGE, IMAGE_TO_IMAGE, TEXT_TO_VIDEO, TEXT_TO_3D, BLENDER_RENDER, GAUSSIAN_SPLATTING, MESH_PROCESSING, TEXTURE_UPSCALE, etc.
  • WorkflowEngine: COMFYUI, BLENDER, UNREAL, GODOT, CUSTOM
  • ModelType: CHECKPOINT, LORA, CONTROLNET, VAE, EMBEDDING, UPSCALER
  • ModelFormat: SAFETENSORS, PYTORCH, ONNX, GGUF, BLEND, GLTF, FBX
  • OutputFileType: IMAGE, VIDEO, AUDIO, MODEL_3D, POINT_CLOUD, ANIMATION, TEXTURE, MATERIAL
  • StorageTier: HOT, WARM, COLD, GLACIER

Common Development Tasks#

Adding a New Generation Type#

  1. Add the type to the GenerationType enum in the Prisma schema (libs/isis/database/prisma/schema.prisma):

    prisma
    enum GenerationType {
      // ... existing types
      YOUR_NEW_TYPE
    }
    
  2. Regenerate the Prisma client:

    bash
    cd libs/isis/database && npx prisma generate
    
  3. Create a new executor in apps/isis/gpu-worker/src/executors/:

    typescript
    import { BaseExecutor } from './base-executor.js';
    
    export class YourNewExecutor extends BaseExecutor {
      async initialize(): Promise<void> {
        /* Load models, etc. */
      }
      async execute(data, allocation, onProgress): Promise<JobResult> {
        /* ... */
      }
      async shutdown(): Promise<void> {
        /* Cleanup */
      }
    }
    
  4. Register the executor in apps/isis/gpu-worker/src/executors/index.ts by adding it to the executor factory.

  5. Create a Docker image if the executor has specific system dependencies: add Dockerfile.your-type in apps/isis/gpu-worker/src/docker/.

  6. Add the job schema in apps/isis/generation-api/src/schemas/ if the generation type has unique parameters.

  7. Add a workflow template in libs/isis/workflows/src/workflows/ if applicable.

Adding a New AI Provider Adapter#

The audio generation library (libs/isis/audio-generation/) demonstrates the provider adapter pattern:

  1. Create a new provider file in the appropriate subdirectory:

    text
    libs/isis/audio-generation/src/generation/your-provider.ts
    
  2. Implement the provider interface matching existing providers (e.g., suno-provider.ts, udio-provider.ts).

  3. Export the provider from the directory's index.ts.

  4. Register the provider in the factory/selector that routes generation requests to providers.

For image/video providers, follow the same pattern in the generation-api's services directory (apps/isis/generation-api/src/services/). The comfyui.service.ts shows how to integrate with external generation backends.

Adding a New Workflow#

  1. Create a JSON workflow definition in the appropriate category directory under libs/isis/workflows/src/workflows/:

    json
    {
      "name": "your-workflow",
      "description": "What this workflow does",
      "engine": "COMFYUI",
      "category": "IMAGE_GENERATION",
      "parameters": [],
      "definition": {}
    }
    
  2. Validate the workflow against the schema defined in libs/isis/workflows/src/schemas/workflow.schema.ts.

  3. The workflow registry API will automatically discover and serve workflows from this directory.


Debugging Tips#

Generation API Not Starting#

  • Check that Redis is running (required for BullMQ job queues)
  • Verify ISIS_DATABASE_URL points to a running PostgreSQL instance
  • The API uses Hono with @hono/zod-openapi; schema validation errors will appear in the response body

GPU Worker Not Processing Jobs#

  • The worker connects to Redis via the REDIS_URL / REDIS_HOST / REDIS_PORT environment variables
  • Jobs are processed from BullMQ queues named by config.queueName
  • Check worker health via the Redis hash worker:<workerId>
  • The worker publishes heartbeats at config.heartbeatInterval seconds; keys expire at 3x the interval
  • Retryable errors (OOM, CUDA, timeout, network) are automatically retried

Model Loading Issues#

  • The GPU worker tracks loaded models in memory via loadedModels Map
  • Model load events are published via @isis/event-publisher
  • Check notifyModelLoaded() calls in executor implementations
  • VRAM usage is tracked per model; check vramUsedMb values

Output Storage Issues#

  • Outputs are stored in MinIO (S3-compatible) locally
  • The storage key format is <storageBucket>/<storageKey>
  • Each output has a hashAlgorithm + hashValue for verification
  • Check the StorageTier for lifecycle status (HOT -> WARM -> COLD -> GLACIER)

Job Queue Monitoring#

  • BullMQ queues can be inspected via Redis commands
  • The GPU worker logs job start, progress, completion, and failure events
  • Check the QueueEvents listener for progress tracking
  • Resource allocation creates working directories under config.executor.workDir