Eve library

Isis Domain Completeness Audit

Guide

Revision 1

On this page

Date: 2026-02-21 Reviewer: Codex (static codebase audit)

Scope#

This audit reviewed the Isis domain for three requirements:

  1. Complex ComfyUI workflow coverage and implementation completeness.
  2. RunPod integration for pay-per-second serverless execution and exposure of Isis flows.
  3. Full feature/workflow exposure via an Isis web app.

Primary paths reviewed:

  • apps/isis/generation-api
  • apps/isis/workflow-registry
  • apps/isis/output-registry
  • apps/isis/gpu-worker
  • libs/isis/workflows
  • libs/isis/ai-providers
  • docker/runpod/comfyui
  • infra/terraform
  • testing/e2e/tests/api/isis
  • tests/integration/flows/isis-yemaya.integration.test.ts

Executive Summary#

The Isis domain is not currently complete against the requested target state.

Key status:

  • Workflow catalog exists (45 workflow JSON files across 8 categories), but end-to-end execution of this catalog is not wired.
  • Generation API job submission persists and emits events, but does not enqueue or execute jobs.
  • RunPod integration exists at adapter and handler layers, but several pay-per-second operational capabilities are stubbed or partially implemented.
  • No dedicated Isis web app exists, and frontend apps do not currently import/use @isis/client in runtime source.

Readiness assessment (estimated):

  • ComfyUI workflow runtime completeness: Low.
  • RunPod serverless production readiness: Medium-Low.
  • Isis web app feature exposure completeness: Not implemented.

Capability Matrix#

Capability Expected Observed Status
Complex workflow inventory Broad category coverage 45 workflows, 8 categories in libs/isis/workflows/src/workflows Partial
Catalog execution wiring Workflow catalog used at runtime Executor defaults are internal fallback IDs; catalog not wired into executor registration Missing
API submit -> execution Job submit triggers processing submit() stores job as pending and returns queue estimate; no queue producer Missing
Worker processing path Shared queue between API and worker GPU worker consumes BullMQ queue (gpu-jobs), but generation API has no BullMQ producer Missing
RunPod invocation Serverless execution path available Adapter exists and is initialized in ComfyUI service Partial
Pay-per-second cost/account insight Accurate cost/history/credits surfaced Cost, credits, and history methods are stubbed/empty Missing
RunPod endpoint provisioning IaC Endpoints created/configured via infra code Secrets/monitoring present, endpoint creation resources absent Partial
Full Isis web app UI exposes all features/interactions/workflows No apps/isis web app; no frontend runtime imports of @isis/client Missing

Findings and Gaps#

P0-1: Job submission is disconnected from execution#

  • Evidence:
    • apps/isis/generation-api/src/services/job.service.ts:508 (submit creates pending job and returns status metadata)
    • apps/isis/generation-api/src/services/job.service.ts:536 (queue position derived from DB records)
    • apps/isis/generation-api/src/routes/jobs.ts:24 (route only calls jobService.submit)
    • apps/isis/gpu-worker/src/workers/gpu-worker.ts:100 (worker consumes BullMQ queue)
    • No BullMQ producer matches in generation API (rg for bullmq|new Queue|Worker\( found no results in apps/isis/generation-api/src).
  • Impact:
    • Submitted jobs can remain pending without entering an executable pipeline.
  • Remediation:
    • Add a queue producer in generation API that enqueues normalized job payloads into the same queue consumed by GPU worker.
    • Add deterministic state transitions: pending -> queued -> running -> completed/failed.
    • Add idempotent retry and dead-letter queue semantics.

P0-2: Workflow runtime and workflow catalog are not integrated#

  • Evidence:
    • apps/isis/generation-api/src/workflows/executor.ts:371 (resolver initialized empty)
    • apps/isis/generation-api/src/workflows/executor.ts:68 (defaults use IDs like flux-dev-basic, wan-basic)
    • libs/isis/workflows/src/utils/registry.ts (45 registered IDs; default executor IDs are not present)
    • apps/isis/generation-api/src/workflows/executor.ts:888 (simulated execution path)
    • apps/isis/generation-api/src/workflows/executor.ts:931 (placeholder URL storage.example.com)
  • Impact:
    • Complex curated workflows are not the runtime source of truth.
    • Runtime can degrade to synthetic output behavior.
  • Remediation:
    • Add bootstrap that registers libs/isis/workflows catalog into executor resolver.
    • Replace hardcoded default IDs with catalog-backed IDs.
    • Fail closed (explicit error) if workflow resolution fails in production.

P0-3: Workflow format contract mismatch across Isis components#

  • Evidence:
    • libs/isis/workflows/src/schemas/workflow.schema.ts:217 (WorkflowNodesSchema is z.record, i.e., object map)
    • libs/isis/workflows/src/workflows/3d/artifact-reconstruction.json:214 ("nodes": { ... } object)
    • apps/isis/workflow-registry/src/services/workflow.service.ts:537 (validator requires definition.nodes array)
    • apps/isis/workflow-registry/src/services/workflow.service.ts:546 (validator requires definition.links array)
  • Impact:
    • Catalog workflows and registry validation are structurally incompatible.
    • Imported complex workflows can fail validation despite being valid under library schema.
  • Remediation:
    • Define one canonical workflow schema for Isis.
    • Add adapter/conversion layer only if dual formats are intentionally supported.
    • Add contract tests to verify all 45 workflows round-trip through registry validation.

P0-4: No Isis web app exposing all interactions/workflows#

  • Evidence:
    • apps/isis contains cli, generation-api, gpu-worker, output-registry, workflow-registry (no web app directory).
    • rg found no runtime source imports of @isis/client in:
      • apps/yemaya/studio-web/src
      • apps/yemaya/studio-desktop/src
      • apps/lilith/web/src
      • apps/oshun/mobile/src
  • Impact:
    • Requirement “every feature/workflow exposed via Isis web app” is currently unmet.
  • Remediation:
    • Create apps/isis/web (or equivalent) and implement full UI surface:
      • Job submission and lifecycle controls
      • Workflow catalog browsing/validation/runs
      • Output/provenance/lineage management
      • Model registration and upload/download flows
      • RunPod provider selection and observability

P1-5: RunPod adapter has production-critical stubs#

  • Evidence:
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:784 (estimateCost returns static placeholder)
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:799 (getCredits returns zeros)
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:812 (getJobHistory returns empty)
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:752 (listModels returns empty)
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:761 (listCustomNodes returns empty)
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:711 (created/started timestamps synthesized via new Date())
  • Impact:
    • Pay-per-second governance (cost tracking, accountability, job history) is incomplete.
  • Remediation:
    • Implement persistent internal accounting from submit/start/complete events.
    • Track worker runtime seconds and map to per-GPU pricing table.
    • Persist provider job metadata and expose queryable history endpoints.

P1-6: RunPod handler supports limited typed modes; full Isis catalog exposure is not explicit#

  • Evidence:
    • docker/runpod/comfyui/handler.py:176 typed input restricted to txt2img, img2img, inpaint, upscale, workflow
    • docker/runpod/comfyui/handler.py:522-530 type validation gates those modes
  • Impact:
    • Domain-specific Isis categories (fractal/sacred-art/avatar/3d/video variants) are only indirectly accessible through generic workflow payloads, with no explicit discovery/exposure contract.
  • Remediation:
    • Add catalog-backed typed endpoints/contracts or explicit workflow-id routing on serverless handler.
    • Publish an endpoint capability map and enforce compatibility checks.

P1-7: GPU worker executors include simulated/placeholder behavior#

  • Evidence:
    • apps/isis/gpu-worker/src/executors/texture-upscale-executor.ts:102 (simulate process; placeholder output)
    • apps/isis/gpu-worker/src/executors/gaussian-splatting-executor.ts:267 (placeholder exports)
    • apps/isis/gpu-worker/src/executors/blender-render-executor.ts:82 (mock mode when Blender absent)
    • apps/isis/gpu-worker/src/executors/mesh-processing-executor.ts:224 (simulate operations/copy placeholder)
  • Impact:
    • Several advanced workflows are not backed by deterministic production execution.
  • Remediation:
    • Gate mock/simulated modes behind explicit non-production flags.
    • Implement real toolchain invocation and output validation checksums.

P1-8: Infrastructure covers secrets/monitoring for RunPod, but endpoint lifecycle provisioning is missing#

  • Evidence:
    • infra/terraform/secrets/runpod.tf stores RunPod API and endpoint IDs as secrets.
    • infra/terraform/cloudwatch/dashboards.tf and infra/terraform/cloudwatch/alarms.tf monitor RunPod metrics.
    • No RunPod endpoint creation resources/modules found under infra/terraform (resource "*runpod" only secrets/monitoring context).
  • Impact:
    • Endpoint setup remains manual and drift-prone; reproducibility is reduced.
  • Remediation:
    • Add explicit endpoint provisioning automation (Terraform external provider/module or managed CI job) with drift detection.

P2-9: Workflow registry is in-memory and demo-auth oriented#

  • Evidence:
    • apps/isis/workflow-registry/src/services/workflow.service.ts:37 (In-Memory Storage (replace with database))
    • apps/isis/workflow-registry/src/app.ts:146 (storageType: 'in-memory')
    • apps/isis/workflow-registry/src/routes/workflows.ts:22 and apps/isis/workflow-registry/src/routes/templates.ts:19 (DEFAULT_OWNER = 'user_demo')
  • Impact:
    • Persistence and multi-tenant ownership controls are not production-ready.
  • Remediation:
    • Move workflows/versions/templates to Postgres.
    • Enforce real auth context for ownership and access control.

P2-10: Output registry still uses demo owner default#

  • Evidence:
    • apps/isis/output-registry/src/routes/outputs.ts:22 (DEFAULT_OWNER = 'user_demo')
  • Impact:
    • Output ownership and access checks can be incorrect under multi-tenant operation.
  • Remediation:
    • Resolve owner from auth middleware/claims and enforce RBAC on output/provenance/lineage operations.

P2-11: Generation API readiness reports queue/storage as OK without backing checks#

  • Evidence:
    • apps/isis/generation-api/src/app.ts:209 (comment: checks would be added later)
    • apps/isis/generation-api/src/app.ts:210 (queue: 'ok')
    • apps/isis/generation-api/src/app.ts:211 (storage: 'ok')
  • Impact:
    • Operational readiness signals can be misleading.
  • Remediation:
    • Add real dependency checks (Redis queue probe, object storage probe) and fail readiness when unavailable.

P2-12: Model upload/download URL generation remains placeholder implementation#

  • Evidence:
    • apps/isis/generation-api/src/services/model.service.ts:604-605 (upload presign placeholder comment)
    • apps/isis/generation-api/src/services/model.service.ts:636 (download presign placeholder comment)
  • Impact:
    • Model management is not fully production-grade.
  • Remediation:
    • Integrate with real storage signing from @oshun/storage and enforce content hash/size verification on completion.

P2-13: Deployment config drift in Docker Compose Isis service paths#

  • Evidence:
    • docker/docker-compose.yml:329 (APP_PATH: apps/isis/api)
    • docker/docker-compose.yml:359 (APP_PATH: apps/isis/generator)
    • Actual directories are apps/isis/generation-api and apps/isis/gpu-worker.
  • Impact:
    • Local/container deployment can fail or run stale targets.
  • Remediation:
    • Correct compose app paths and add CI check validating compose APP_PATH values against filesystem.

P2-14: Test coverage does not validate real end-to-end execution and RunPod behavior#

  • Evidence:
    • E2E coverage is API-centric: testing/e2e/tests/api/isis/jobs.e2e.ts, models.e2e.ts, etc.
    • Integration flow uses mocked event bus fixtures: tests/integration/flows/isis-yemaya.integration.test.ts.
  • Impact:
    • Critical gaps (queue wiring, RunPod live execution, catalog compatibility) are not automatically detected.
  • Remediation:
    • Add contract/integration tests for:
      • submit -> queue -> worker -> output registry end-to-end
      • RunPod sandbox endpoint execution and status/cost accounting
      • workflow catalog compatibility across loader, registry, executor

Proposed Remediation Plan#

Phase 0 (0-2 weeks): Make execution real and deterministic#

  1. Wire generation API to BullMQ producer with consistent queue naming.
  2. Connect job lifecycle updates from worker callbacks.
  3. Integrate workflow catalog registration into executor startup.
  4. Fix schema mismatch between catalog and registry validator.
  5. Fix Docker Compose Isis app paths.

Exit criteria:

  • A submitted job transitions to completed with non-placeholder output for at least one workflow in each major category (video, 3d, avatar, fractal, utility).

Phase 1 (2-4 weeks): Productionize RunPod serverless operations#

  1. Implement real cost accounting and per-job history persistence.
  2. Add explicit capability mapping from Isis workflow IDs to RunPod handlers/endpoints.
  3. Add endpoint provisioning automation and drift checks.
  4. Add readiness and SLO metrics for queue/storage/provider health.

Exit criteria:

  • RunPod jobs have auditable runtime seconds, calculated cost, and queryable historical records.

Phase 2 (3-6 weeks): Build Isis web app for full feature exposure#

  1. Create apps/isis/web with authenticated multi-tenant UI.
  2. Expose all Isis surfaces (jobs, workflows, outputs/lineage, models, provider telemetry).
  3. Add category and workflow-ID based run experiences for all supported catalog entries.

Exit criteria:

  • Every API capability in @isis/client has UI coverage and route-level integration tests.

Phase 3 (ongoing): Hardening and confidence#

  1. Replace all simulated executor paths for production profiles.
  2. Add full E2E test pipeline in CI using ephemeral queue/db/storage and a RunPod test endpoint.
  3. Add release gates that block deploy on catalog-contract or provider-health regression.

Risk Register#

  • High risk: Jobs accepted but not executed due to missing queue producer.
  • High risk: Workflow contract mismatch causes silent feature non-availability.
  • Medium risk: Cost/accountability blind spots for serverless usage.
  • Medium risk: Missing web app blocks stakeholder-facing completeness objective.

Conclusion#

The Isis domain has substantial foundational assets (workflow corpus, provider adapters, service decomposition), but it does not yet meet the stated requirement of complete complex-workflow implementation, pay-per-second RunPod serverless readiness, and full Isis web app feature exposure. The highest-value next step is to close the submit-to-execution wiring and workflow schema integration before expanding UI and operational polish.