Isis is the Generative AI Factory domain. It provides GPU-accelerated generation services for images, video, 3D models, audio, and more. It manages ComfyUI workflows, GPU worker orchestration, model registries, output provenance tracking, and multi-format generation pipelines.
Prerequisites#
Required Software#
| Software | Minimum Version | Purpose |
|---|---|---|
| Node.js | >= 18.0.0 | Runtime for API services |
| pnpm | >= 9.0.0 | Package manager (monorepo workspace) |
| Docker | >= 24.0 | Infrastructure and GPU worker images |
| TypeScript | ^6.0.3 | Type checking and compilation |
For GPU Worker Development#
| Software | Version | Purpose |
|---|---|---|
| NVIDIA Driver | >= 535 | GPU access for workers |
| CUDA Toolkit | >= 12.0 | GPU compute for AI models |
| nvidia-docker | >= 2.0 | Docker GPU passthrough |
| Blender | >= 4.0 | Blender render executor |
GPU hardware is only required for running gpu-worker executors. All other
services (generation-api, output-registry, workflow-registry, cli) run on
standard hardware.
Project Structure#
apps/isis/
generation-api/ # Main API - job submission, status tracking, model management
gpu-worker/ # GPU job processor (BullMQ worker with executor plugins)
output-registry/ # Output file management API (storage, retrieval, retention)
workflow-registry/ # Workflow template management API
cli/ # Command-line interface for generation, jobs, workflows
libs/isis/
client/ # TypeScript API client (@isis/client)
database/ # Prisma schema and generated client
audio-generation/ # Audio/voice/music generation (ElevenLabs, Suno, Udio)
workflows/ # Workflow JSON definitions (ComfyUI, Blender, etc.)
event-publisher/ # Cross-domain event publisher (@isis/event-publisher)
Workflow Library Categories#
The libs/isis/workflows/ directory contains pre-built workflow definitions:
sacred-art/- Mandala, thangka, yantra, celtic knotwork generatorsfractal/- Mandelbrot, Julia, flame fractal, Lorenz attractoravatar/- Face swap, cosmetic layers, consistent characterutility/- Background removal, style transfer, batch operationsnft/- Rarity enhancement workflows
Local Development Setup#
1. Prerequisites#
# Required
docker --version
docker compose version
pnpm --version
GPU acceleration is optional for local startup, but if you want to run the worker against local GPUs you need NVIDIA drivers and NVIDIA Container Toolkit installed on the host.
2. Start Isis Services with Compose (GPU Profile)#
# From monorepo root (full Isis profile including GPU worker)
docker compose -f docker/docker-compose.yml --profile gpu up -d \
postgres redis qdrant isis-api isis-generator
If you do not have local GPU support, start only the API and its dependencies:
docker compose -f docker/docker-compose.yml --profile gpu up -d \
postgres redis qdrant isis-api
Default API host port in this compose stack is http://localhost:3003.
3. Validate Compose Wiring and Smoke Health#
# Validate APP_PATH references used by Isis compose services
bash scripts/isis/validate_compose_app_paths.sh
# Compose smoke test (config validation only; fast)
CONFIG_ONLY=true scripts/isis/smoke_compose_health.sh
# Full smoke test: boots Isis compose services and checks /health + /ready
# Note: first run can be slow because Docker images may need to build/pull.
scripts/isis/smoke_compose_health.sh
4. Verify Health Endpoints Manually#
curl -fsS http://localhost:${ISIS_API_PORT:-3003}/health
# /ready may return 200 (ready) or 503 (degraded/unready) depending on optional dependencies.
curl -s -o /tmp/isis-ready.json -w "%{http_code}\n" \
http://localhost:${ISIS_API_PORT:-3003}/ready
5. Environment Variables (Local Reference)#
| Variable | Default | Used By |
|---|---|---|
ISIS_DATABASE_URL |
(from root .env) |
database lib |
REDIS_URL |
redis://localhost:6379 |
generation-api, gpu-worker |
ISIS_QUEUE_NAME |
isis:jobs:submit |
generation-api |
ISIS_QUEUE_REDIS_URL |
(optional in dev) | generation-api |
ISIS_QUEUE_PRODUCER_ENABLED |
true |
generation-api |
ISIS_QUEUE_PRODUCER_CANARY_ENABLED |
false |
generation-api |
ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE |
0 |
generation-api |
ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS |
(optional CSV) | generation-api |
REDIS_HOST |
localhost |
gpu-worker |
REDIS_PORT |
6379 |
gpu-worker |
PORT |
3000 |
generation-api |
HOST |
localhost |
generation-api |
ISIS_MODEL_MALWARE_SCAN_ENABLED |
true |
generation-api |
ISIS_MODEL_ABANDONED_UPLOAD_RETENTION_HOURS |
24 |
generation-api |
For production-like startup, generation-api requires queue configuration
(ISIS_QUEUE_NAME and either ISIS_QUEUE_REDIS_URL or REDIS_URL) when queue
dispatch is active:
ISIS_QUEUE_PRODUCER_ENABLED=true, orISIS_QUEUE_PRODUCER_ENABLED=falsewithISIS_QUEUE_PRODUCER_CANARY_ENABLED=trueandISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE>0.
6. Verify Canary SLO and Cost Gate (Release Readiness)#
ISIS_DATABASE_URL=postgres://... \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
scripts/isis/verify_canary_slo_cost.sh
To enforce gate failure in automation:
ISIS_DATABASE_URL=postgres://... \
FAIL_ON_BREACH=true \
scripts/isis/verify_canary_slo_cost.sh
Guide and threshold details:
docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md
7. Advance Canary Ramp with Auto-Rollback Decisioning#
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
Auto-rollback (decision + non-zero exit) on threshold failure:
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=true \
scripts/isis/advance_queue_canary_rollout.sh
Guide and callback integration details:
docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md
8. Run Release Game-Day Harness (Pre-Cutover)#
ISIS_DATABASE_URL=postgres://... \
ISIS_API_BASE_URL=https://<isis-api-host> \
CURRENT_PERCENTAGE=10 \
STRICT=true \
scripts/isis/run_release_gameday.sh
Plan and reporting templates:
docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.mddocs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md
9. Run Post-Cutover Validation and Sign-Off Harness#
ISIS_API_BASE_URL=https://<isis-api-host> \
ISIS_DATABASE_URL=postgres://... \
STRICT=true \
scripts/isis/run_post_cutover_validation.sh
Checklist and sign-off templates:
docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.mddocs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md
10. Install Dependencies (Direct Local Runs)#
pnpm install
11. Seed the Database (optional for direct runs)#
cd libs/isis/database
npx tsx prisma/seed.ts # Production-like seed data
npx tsx prisma/seed-test.ts # Test fixtures
Running Services#
Generation API#
The main entry point for submitting generation jobs, querying status, and managing models. Built with Hono and Zod OpenAPI.
# Development (hot reload)
cd apps/isis/generation-api
pnpm dev # tsx watch src/index.ts
# Via Nx
pnpm nx serve @isis/generation-api
The API exposes:
POST /jobs- Submit generation jobsGET /jobs/:id- Check job statusGET /models- List available modelsPOST /models- Register new models
GPU Worker#
The GPU worker pulls jobs from BullMQ queues and routes them to the appropriate executor. Each executor handles a specific generation type.
# Development (no GPU needed for the service itself)
pnpm nx serve @isis/gpu-worker
# Specific configuration
pnpm nx serve @isis/gpu-worker --configuration=development
Available executors:
| Executor | File | Purpose |
|---|---|---|
| Blender Render | blender-render-executor.ts |
Blender scene rendering |
| Mesh Processing | mesh-processing-executor.ts |
3D mesh operations |
| Texture Upscale | texture-upscale-executor.ts |
AI texture upscaling |
Output Registry#
Manages generated output files with storage lifecycle, retention policies, and provenance tracking.
pnpm nx serve @isis/output-registry
Workflow Registry#
Manages workflow templates and versioning.
pnpm nx serve @isis/workflow-registry
CLI#
# Development
pnpm nx dev @isis/cli
# Available commands
isis generate # Submit a generation job
isis jobs # List/manage jobs
isis outputs # Browse generated outputs
isis workflows # Manage workflows
isis auth # Authentication
isis health # Service health check
isis config # Configuration management
isis jobs triage # Operator stuck-job and DLQ triage
isis jobs replay # Replay dead-letter outbox entries
Building#
Build All Isis Projects#
pnpm nx run-many --target=build --projects=tag:scope:isis
Build Individual Projects#
# Generation API (tsc)
pnpm nx build @isis/generation-api
# GPU Worker (esbuild, bundled)
pnpm nx build @isis/gpu-worker
pnpm nx build @isis/gpu-worker --configuration=production # Minified
# Client library
pnpm nx build @isis/client
Build GPU Worker Docker Images#
The gpu-worker has specialized Docker images for each executor type:
pnpm nx docker-build @isis/gpu-worker
This builds four images:
isis/gpu-worker:base- Base image with shared dependenciesisis/gpu-worker:texture-upscale- Texture upscaling executorisis/gpu-worker:mesh-processing- Mesh processing executorisis/gpu-worker:blender-render- Blender rendering executor
Dockerfiles are located in apps/isis/gpu-worker/src/docker/.
Testing#
Test Runner Configuration#
All Isis apps use Vitest with the @nx/vite:test executor.
Vitest configuration (vitest.config.ts):
export default defineConfig({
test: {
globals: true,
environment: 'node',
include: ['src/**/*.{test,spec}.ts'],
coverage: {
provider: 'v8',
reporter: ['text', 'json', 'html'],
exclude: ['node_modules/', 'dist/', '**/*.d.ts'],
},
},
});
Running Tests#
# All Isis projects
pnpm nx run-many --target=test --projects=tag:scope:isis
# Individual projects
pnpm nx test @isis/generation-api
pnpm nx test @isis/gpu-worker
pnpm nx test @isis/output-registry
pnpm nx test @isis/workflow-registry
pnpm nx test @isis/client
Testing Without a GPU#
The GPU worker's executors can be tested without actual GPU hardware:
-
Unit tests - All executors extend
BaseExecutorwhich defines a standard interface. Mock the executor in tests:typescriptimport { vi } from 'vitest'; const mockExecutor = { initialize: vi.fn().mockResolvedValue(undefined), execute: vi.fn().mockResolvedValue({ status: 'completed', outputs: [] }), shutdown: vi.fn().mockResolvedValue(undefined), }; -
Integration tests - The
GPUWorkerclass can be instantiated withgpuMonitoring: falsein the config. GPU info calls return placeholder data when monitoring is disabled. -
Mock worker approach - Set the worker config's
gpuDevicesto[0]and disable cleanup. The worker will process jobs through the executor pipeline without requiring real GPU hardware.
Linting#
pnpm nx run-many --target=lint --projects=tag:scope:isis
pnpm nx lint @isis/generation-api
Database Schema#
The Isis database (libs/isis/database/prisma/schema.prisma) uses PostgreSQL
with the following key models:
| Model | Purpose |
|---|---|
GenerationJob |
Job records with type, status, prompt, model, worker assignment |
Workflow |
Workflow definitions with versioning, parameters, engine type |
WorkflowVersion |
Workflow version history with definition snapshots |
WorkflowTemplate |
Pre-built workflow templates |
WorkflowStar |
User favorites on workflows |
GeneratedOutput |
Output files with storage location, dimensions, lineage |
LineageEdge |
Derivation relationships between outputs |
Provenance |
Full provenance records (model, LoRAs, seed, GPU, cost) |
ModelRegistry |
AI model/checkpoint registry |
RetentionPolicy |
Storage lifecycle rules |
GpuWorker |
GPU worker registration and health tracking |
AuditLog |
Change tracking |
Key Enums#
GenerationType: TEXT_TO_IMAGE, IMAGE_TO_IMAGE, TEXT_TO_VIDEO, TEXT_TO_3D, BLENDER_RENDER, GAUSSIAN_SPLATTING, MESH_PROCESSING, TEXTURE_UPSCALE, etc.WorkflowEngine: COMFYUI, BLENDER, UNREAL, GODOT, CUSTOMModelType: CHECKPOINT, LORA, CONTROLNET, VAE, EMBEDDING, UPSCALERModelFormat: SAFETENSORS, PYTORCH, ONNX, GGUF, BLEND, GLTF, FBXOutputFileType: IMAGE, VIDEO, AUDIO, MODEL_3D, POINT_CLOUD, ANIMATION, TEXTURE, MATERIALStorageTier: HOT, WARM, COLD, GLACIER
Common Development Tasks#
Adding a New Generation Type#
-
Add the type to the
GenerationTypeenum in the Prisma schema (libs/isis/database/prisma/schema.prisma):prismaenum GenerationType { // ... existing types YOUR_NEW_TYPE } -
Regenerate the Prisma client:
bashcd libs/isis/database && npx prisma generate -
Create a new executor in
apps/isis/gpu-worker/src/executors/:typescriptimport { BaseExecutor } from './base-executor.js'; export class YourNewExecutor extends BaseExecutor { async initialize(): Promise<void> { /* Load models, etc. */ } async execute(data, allocation, onProgress): Promise<JobResult> { /* ... */ } async shutdown(): Promise<void> { /* Cleanup */ } } -
Register the executor in
apps/isis/gpu-worker/src/executors/index.tsby adding it to the executor factory. -
Create a Docker image if the executor has specific system dependencies: add
Dockerfile.your-typeinapps/isis/gpu-worker/src/docker/. -
Add the job schema in
apps/isis/generation-api/src/schemas/if the generation type has unique parameters. -
Add a workflow template in
libs/isis/workflows/src/workflows/if applicable.
Adding a New AI Provider Adapter#
The audio generation library (libs/isis/audio-generation/) demonstrates the
provider adapter pattern:
-
Create a new provider file in the appropriate subdirectory:
textlibs/isis/audio-generation/src/generation/your-provider.ts -
Implement the provider interface matching existing providers (e.g.,
suno-provider.ts,udio-provider.ts). -
Export the provider from the directory's
index.ts. -
Register the provider in the factory/selector that routes generation requests to providers.
For image/video providers, follow the same pattern in the generation-api's
services directory (apps/isis/generation-api/src/services/). The
comfyui.service.ts shows how to integrate with external generation backends.
Adding a New Workflow#
-
Create a JSON workflow definition in the appropriate category directory under
libs/isis/workflows/src/workflows/:json{ "name": "your-workflow", "description": "What this workflow does", "engine": "COMFYUI", "category": "IMAGE_GENERATION", "parameters": [], "definition": {} } -
Validate the workflow against the schema defined in
libs/isis/workflows/src/schemas/workflow.schema.ts. -
The workflow registry API will automatically discover and serve workflows from this directory.
Debugging Tips#
Generation API Not Starting#
- Check that Redis is running (required for BullMQ job queues)
- Verify
ISIS_DATABASE_URLpoints to a running PostgreSQL instance - The API uses Hono with
@hono/zod-openapi; schema validation errors will appear in the response body
GPU Worker Not Processing Jobs#
- The worker connects to Redis via the
REDIS_URL/REDIS_HOST/REDIS_PORTenvironment variables - Jobs are processed from BullMQ queues named by
config.queueName - Check worker health via the Redis hash
worker:<workerId> - The worker publishes heartbeats at
config.heartbeatIntervalseconds; keys expire at 3x the interval - Retryable errors (OOM, CUDA, timeout, network) are automatically retried
Model Loading Issues#
- The GPU worker tracks loaded models in memory via
loadedModelsMap - Model load events are published via
@isis/event-publisher - Check
notifyModelLoaded()calls in executor implementations - VRAM usage is tracked per model; check
vramUsedMbvalues
Output Storage Issues#
- Outputs are stored in MinIO (S3-compatible) locally
- The storage key format is
<storageBucket>/<storageKey> - Each output has a
hashAlgorithm+hashValuefor verification - Check the
StorageTierfor lifecycle status (HOT -> WARM -> COLD -> GLACIER)
Job Queue Monitoring#
- BullMQ queues can be inspected via Redis commands
- The GPU worker logs job start, progress, completion, and failure events
- Check the
QueueEventslistener for progress tracking - Resource allocation creates working directories under
config.executor.workDir