# Isis Domain - Developer Guide

Isis is the Generative AI Factory domain. It provides GPU-accelerated generation
services for images, video, 3D models, audio, and more. It manages ComfyUI
workflows, GPU worker orchestration, model registries, output provenance
tracking, and multi-format generation pipelines.

---

## Prerequisites

### Required Software

| Software   | Minimum Version | Purpose                              |
| ---------- | --------------- | ------------------------------------ |
| Node.js    | >= 18.0.0       | Runtime for API services             |
| pnpm       | >= 9.0.0        | Package manager (monorepo workspace) |
| Docker     | >= 24.0         | Infrastructure and GPU worker images |
| TypeScript | ^6.0.3          | Type checking and compilation        |

### For GPU Worker Development

| Software      | Version | Purpose                   |
| ------------- | ------- | ------------------------- |
| NVIDIA Driver | >= 535  | GPU access for workers    |
| CUDA Toolkit  | >= 12.0 | GPU compute for AI models |
| nvidia-docker | >= 2.0  | Docker GPU passthrough    |
| Blender       | >= 4.0  | Blender render executor   |

GPU hardware is only required for running `gpu-worker` executors. All other
services (generation-api, output-registry, workflow-registry, cli) run on
standard hardware.

---

## Project Structure

```
apps/isis/
  generation-api/     # Main API - job submission, status tracking, model management
  gpu-worker/         # GPU job processor (BullMQ worker with executor plugins)
  output-registry/    # Output file management API (storage, retrieval, retention)
  workflow-registry/  # Workflow template management API
  cli/                # Command-line interface for generation, jobs, workflows

libs/isis/
  client/             # TypeScript API client (@isis/client)
  database/           # Prisma schema and generated client
  audio-generation/   # Audio/voice/music generation (ElevenLabs, Suno, Udio)
  workflows/          # Workflow JSON definitions (ComfyUI, Blender, etc.)
  event-publisher/    # Cross-domain event publisher (@isis/event-publisher)
```

### Workflow Library Categories

The `libs/isis/workflows/` directory contains pre-built workflow definitions:

- `sacred-art/` - Mandala, thangka, yantra, celtic knotwork generators
- `fractal/` - Mandelbrot, Julia, flame fractal, Lorenz attractor
- `avatar/` - Face swap, cosmetic layers, consistent character
- `utility/` - Background removal, style transfer, batch operations
- `nft/` - Rarity enhancement workflows

---

## Local Development Setup

### 1. Prerequisites

```bash
# Required
docker --version
docker compose version
pnpm --version
```

GPU acceleration is optional for local startup, but if you want to run the
worker against local GPUs you need NVIDIA drivers and NVIDIA Container Toolkit
installed on the host.

### 2. Start Isis Services with Compose (GPU Profile)

```bash
# From monorepo root (full Isis profile including GPU worker)
docker compose -f docker/docker-compose.yml --profile gpu up -d \
  postgres redis qdrant isis-api isis-generator
```

If you do not have local GPU support, start only the API and its dependencies:

```bash
docker compose -f docker/docker-compose.yml --profile gpu up -d \
  postgres redis qdrant isis-api
```

Default API host port in this compose stack is `http://localhost:3003`.

### 3. Validate Compose Wiring and Smoke Health

```bash
# Validate APP_PATH references used by Isis compose services
bash scripts/isis/validate_compose_app_paths.sh

# Compose smoke test (config validation only; fast)
CONFIG_ONLY=true scripts/isis/smoke_compose_health.sh

# Full smoke test: boots Isis compose services and checks /health + /ready
# Note: first run can be slow because Docker images may need to build/pull.
scripts/isis/smoke_compose_health.sh
```

### 4. Verify Health Endpoints Manually

```bash
curl -fsS http://localhost:${ISIS_API_PORT:-3003}/health

# /ready may return 200 (ready) or 503 (degraded/unready) depending on optional dependencies.
curl -s -o /tmp/isis-ready.json -w "%{http_code}\n" \
  http://localhost:${ISIS_API_PORT:-3003}/ready
```

### 5. Environment Variables (Local Reference)

| Variable                                      | Default                  | Used By                    |
| --------------------------------------------- | ------------------------ | -------------------------- |
| `ISIS_DATABASE_URL`                           | (from root `.env`)       | database lib               |
| `REDIS_URL`                                   | `redis://localhost:6379` | generation-api, gpu-worker |
| `ISIS_QUEUE_NAME`                             | `isis:jobs:submit`       | generation-api             |
| `ISIS_QUEUE_REDIS_URL`                        | (optional in dev)        | generation-api             |
| `ISIS_QUEUE_PRODUCER_ENABLED`                 | `true`                   | generation-api             |
| `ISIS_QUEUE_PRODUCER_CANARY_ENABLED`          | `false`                  | generation-api             |
| `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE`       | `0`                      | generation-api             |
| `ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS`      | (optional CSV)           | generation-api             |
| `REDIS_HOST`                                  | `localhost`              | gpu-worker                 |
| `REDIS_PORT`                                  | `6379`                   | gpu-worker                 |
| `PORT`                                        | `3000`                   | generation-api             |
| `HOST`                                        | `localhost`              | generation-api             |
| `ISIS_MODEL_MALWARE_SCAN_ENABLED`             | `true`                   | generation-api             |
| `ISIS_MODEL_ABANDONED_UPLOAD_RETENTION_HOURS` | `24`                     | generation-api             |

For production-like startup, generation-api requires queue configuration
(`ISIS_QUEUE_NAME` and either `ISIS_QUEUE_REDIS_URL` or `REDIS_URL`) when queue
dispatch is active:

- `ISIS_QUEUE_PRODUCER_ENABLED=true`, or
- `ISIS_QUEUE_PRODUCER_ENABLED=false` with
  `ISIS_QUEUE_PRODUCER_CANARY_ENABLED=true` and
  `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE>0`.

### 6. Verify Canary SLO and Cost Gate (Release Readiness)

```bash
ISIS_DATABASE_URL=postgres://... \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
scripts/isis/verify_canary_slo_cost.sh
```

To enforce gate failure in automation:

```bash
ISIS_DATABASE_URL=postgres://... \
FAIL_ON_BREACH=true \
scripts/isis/verify_canary_slo_cost.sh
```

Guide and threshold details:
`docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md`

### 7. Advance Canary Ramp with Auto-Rollback Decisioning

```bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
```

Auto-rollback (decision + non-zero exit) on threshold failure:

```bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=true \
scripts/isis/advance_queue_canary_rollout.sh
```

Guide and callback integration details:
`docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md`

### 8. Run Release Game-Day Harness (Pre-Cutover)

```bash
ISIS_DATABASE_URL=postgres://... \
ISIS_API_BASE_URL=https://<isis-api-host> \
CURRENT_PERCENTAGE=10 \
STRICT=true \
scripts/isis/run_release_gameday.sh
```

Plan and reporting templates:

- `docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md`
- `docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md`

### 9. Run Post-Cutover Validation and Sign-Off Harness

```bash
ISIS_API_BASE_URL=https://<isis-api-host> \
ISIS_DATABASE_URL=postgres://... \
STRICT=true \
scripts/isis/run_post_cutover_validation.sh
```

Checklist and sign-off templates:

- `docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md`
- `docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md`

### 10. Install Dependencies (Direct Local Runs)

```bash
pnpm install
```

### 11. Seed the Database (optional for direct runs)

```bash
cd libs/isis/database
npx tsx prisma/seed.ts       # Production-like seed data
npx tsx prisma/seed-test.ts  # Test fixtures
```

---

## Running Services

### Generation API

The main entry point for submitting generation jobs, querying status, and
managing models. Built with Hono and Zod OpenAPI.

```bash
# Development (hot reload)
cd apps/isis/generation-api
pnpm dev   # tsx watch src/index.ts

# Via Nx
pnpm nx serve @isis/generation-api
```

The API exposes:

- `POST /jobs` - Submit generation jobs
- `GET /jobs/:id` - Check job status
- `GET /models` - List available models
- `POST /models` - Register new models

### GPU Worker

The GPU worker pulls jobs from BullMQ queues and routes them to the appropriate
executor. Each executor handles a specific generation type.

```bash
# Development (no GPU needed for the service itself)
pnpm nx serve @isis/gpu-worker

# Specific configuration
pnpm nx serve @isis/gpu-worker --configuration=development
```

**Available executors:**

| Executor        | File                          | Purpose                 |
| --------------- | ----------------------------- | ----------------------- |
| Blender Render  | `blender-render-executor.ts`  | Blender scene rendering |
| Mesh Processing | `mesh-processing-executor.ts` | 3D mesh operations      |
| Texture Upscale | `texture-upscale-executor.ts` | AI texture upscaling    |

### Output Registry

Manages generated output files with storage lifecycle, retention policies, and
provenance tracking.

```bash
pnpm nx serve @isis/output-registry
```

### Workflow Registry

Manages workflow templates and versioning.

```bash
pnpm nx serve @isis/workflow-registry
```

### CLI

```bash
# Development
pnpm nx dev @isis/cli

# Available commands
isis generate   # Submit a generation job
isis jobs       # List/manage jobs
isis outputs    # Browse generated outputs
isis workflows  # Manage workflows
isis auth       # Authentication
isis health     # Service health check
isis config     # Configuration management
isis jobs triage # Operator stuck-job and DLQ triage
isis jobs replay # Replay dead-letter outbox entries
```

---

## Building

### Build All Isis Projects

```bash
pnpm nx run-many --target=build --projects=tag:scope:isis
```

### Build Individual Projects

```bash
# Generation API (tsc)
pnpm nx build @isis/generation-api

# GPU Worker (esbuild, bundled)
pnpm nx build @isis/gpu-worker
pnpm nx build @isis/gpu-worker --configuration=production  # Minified

# Client library
pnpm nx build @isis/client
```

### Build GPU Worker Docker Images

The gpu-worker has specialized Docker images for each executor type:

```bash
pnpm nx docker-build @isis/gpu-worker
```

This builds four images:

- `isis/gpu-worker:base` - Base image with shared dependencies
- `isis/gpu-worker:texture-upscale` - Texture upscaling executor
- `isis/gpu-worker:mesh-processing` - Mesh processing executor
- `isis/gpu-worker:blender-render` - Blender rendering executor

Dockerfiles are located in `apps/isis/gpu-worker/src/docker/`.

---

## Testing

### Test Runner Configuration

All Isis apps use Vitest with the `@nx/vite:test` executor.

Vitest configuration (`vitest.config.ts`):

```typescript
export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    include: ['src/**/*.{test,spec}.ts'],
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
      exclude: ['node_modules/', 'dist/', '**/*.d.ts'],
    },
  },
});
```

### Running Tests

```bash
# All Isis projects
pnpm nx run-many --target=test --projects=tag:scope:isis

# Individual projects
pnpm nx test @isis/generation-api
pnpm nx test @isis/gpu-worker
pnpm nx test @isis/output-registry
pnpm nx test @isis/workflow-registry
pnpm nx test @isis/client
```

### Testing Without a GPU

The GPU worker's executors can be tested without actual GPU hardware:

1. **Unit tests** - All executors extend `BaseExecutor` which defines a standard
   interface. Mock the executor in tests:

   ```typescript
   import { vi } from 'vitest';

   const mockExecutor = {
     initialize: vi.fn().mockResolvedValue(undefined),
     execute: vi.fn().mockResolvedValue({ status: 'completed', outputs: [] }),
     shutdown: vi.fn().mockResolvedValue(undefined),
   };
   ```

2. **Integration tests** - The `GPUWorker` class can be instantiated with
   `gpuMonitoring: false` in the config. GPU info calls return placeholder data
   when monitoring is disabled.

3. **Mock worker approach** - Set the worker config's `gpuDevices` to `[0]` and
   disable cleanup. The worker will process jobs through the executor pipeline
   without requiring real GPU hardware.

### Linting

```bash
pnpm nx run-many --target=lint --projects=tag:scope:isis
pnpm nx lint @isis/generation-api
```

---

## Database Schema

The Isis database (`libs/isis/database/prisma/schema.prisma`) uses PostgreSQL
with the following key models:

| Model              | Purpose                                                         |
| ------------------ | --------------------------------------------------------------- |
| `GenerationJob`    | Job records with type, status, prompt, model, worker assignment |
| `Workflow`         | Workflow definitions with versioning, parameters, engine type   |
| `WorkflowVersion`  | Workflow version history with definition snapshots              |
| `WorkflowTemplate` | Pre-built workflow templates                                    |
| `WorkflowStar`     | User favorites on workflows                                     |
| `GeneratedOutput`  | Output files with storage location, dimensions, lineage         |
| `LineageEdge`      | Derivation relationships between outputs                        |
| `Provenance`       | Full provenance records (model, LoRAs, seed, GPU, cost)         |
| `ModelRegistry`    | AI model/checkpoint registry                                    |
| `RetentionPolicy`  | Storage lifecycle rules                                         |
| `GpuWorker`        | GPU worker registration and health tracking                     |
| `AuditLog`         | Change tracking                                                 |

### Key Enums

- `GenerationType`: TEXT_TO_IMAGE, IMAGE_TO_IMAGE, TEXT_TO_VIDEO, TEXT_TO_3D,
  BLENDER_RENDER, GAUSSIAN_SPLATTING, MESH_PROCESSING, TEXTURE_UPSCALE, etc.
- `WorkflowEngine`: COMFYUI, BLENDER, UNREAL, GODOT, CUSTOM
- `ModelType`: CHECKPOINT, LORA, CONTROLNET, VAE, EMBEDDING, UPSCALER
- `ModelFormat`: SAFETENSORS, PYTORCH, ONNX, GGUF, BLEND, GLTF, FBX
- `OutputFileType`: IMAGE, VIDEO, AUDIO, MODEL_3D, POINT_CLOUD, ANIMATION,
  TEXTURE, MATERIAL
- `StorageTier`: HOT, WARM, COLD, GLACIER

---

## Common Development Tasks

### Adding a New Generation Type

1. Add the type to the `GenerationType` enum in the Prisma schema
   (`libs/isis/database/prisma/schema.prisma`):

   ```prisma
   enum GenerationType {
     // ... existing types
     YOUR_NEW_TYPE
   }
   ```

2. Regenerate the Prisma client:

   ```bash
   cd libs/isis/database && npx prisma generate
   ```

3. Create a new executor in `apps/isis/gpu-worker/src/executors/`:

   ```typescript
   import { BaseExecutor } from './base-executor.js';

   export class YourNewExecutor extends BaseExecutor {
     async initialize(): Promise<void> {
       /* Load models, etc. */
     }
     async execute(data, allocation, onProgress): Promise<JobResult> {
       /* ... */
     }
     async shutdown(): Promise<void> {
       /* Cleanup */
     }
   }
   ```

4. Register the executor in `apps/isis/gpu-worker/src/executors/index.ts` by
   adding it to the executor factory.

5. Create a Docker image if the executor has specific system dependencies: add
   `Dockerfile.your-type` in `apps/isis/gpu-worker/src/docker/`.

6. Add the job schema in `apps/isis/generation-api/src/schemas/` if the
   generation type has unique parameters.

7. Add a workflow template in `libs/isis/workflows/src/workflows/` if
   applicable.

### Adding a New AI Provider Adapter

The audio generation library (`libs/isis/audio-generation/`) demonstrates the
provider adapter pattern:

1. Create a new provider file in the appropriate subdirectory:

   ```
   libs/isis/audio-generation/src/generation/your-provider.ts
   ```

2. Implement the provider interface matching existing providers (e.g.,
   `suno-provider.ts`, `udio-provider.ts`).

3. Export the provider from the directory's `index.ts`.

4. Register the provider in the factory/selector that routes generation requests
   to providers.

For image/video providers, follow the same pattern in the generation-api's
services directory (`apps/isis/generation-api/src/services/`). The
`comfyui.service.ts` shows how to integrate with external generation backends.

### Adding a New Workflow

1. Create a JSON workflow definition in the appropriate category directory under
   `libs/isis/workflows/src/workflows/`:

   ```json
   {
     "name": "your-workflow",
     "description": "What this workflow does",
     "engine": "COMFYUI",
     "category": "IMAGE_GENERATION",
     "parameters": [],
     "definition": {}
   }
   ```

2. Validate the workflow against the schema defined in
   `libs/isis/workflows/src/schemas/workflow.schema.ts`.

3. The workflow registry API will automatically discover and serve workflows
   from this directory.

---

## Debugging Tips

### Generation API Not Starting

- Check that Redis is running (required for BullMQ job queues)
- Verify `ISIS_DATABASE_URL` points to a running PostgreSQL instance
- The API uses Hono with `@hono/zod-openapi`; schema validation errors will
  appear in the response body

### GPU Worker Not Processing Jobs

- The worker connects to Redis via the `REDIS_URL` / `REDIS_HOST` / `REDIS_PORT`
  environment variables
- Jobs are processed from BullMQ queues named by `config.queueName`
- Check worker health via the Redis hash `worker:<workerId>`
- The worker publishes heartbeats at `config.heartbeatInterval` seconds; keys
  expire at 3x the interval
- Retryable errors (OOM, CUDA, timeout, network) are automatically retried

### Model Loading Issues

- The GPU worker tracks loaded models in memory via `loadedModels` Map
- Model load events are published via `@isis/event-publisher`
- Check `notifyModelLoaded()` calls in executor implementations
- VRAM usage is tracked per model; check `vramUsedMb` values

### Output Storage Issues

- Outputs are stored in MinIO (S3-compatible) locally
- The storage key format is `<storageBucket>/<storageKey>`
- Each output has a `hashAlgorithm` + `hashValue` for verification
- Check the `StorageTier` for lifecycle status (HOT -> WARM -> COLD -> GLACIER)

### Job Queue Monitoring

- BullMQ queues can be inspected via Redis commands
- The GPU worker logs job start, progress, completion, and failure events
- Check the `QueueEvents` listener for progress tracking
- Resource allocation creates working directories under
  `config.executor.workDir`
