Eve library

ADR-0003: RunPod Serverless Execution Strategy and Governance

ADR

Revision 1

On this page

Status: Accepted
Date: 2026-02-21
Authors: Isis ML Platform, Isis Runtime Engineering
Reviewers: @oshun/ml-team, @oshun/devops, @oshun/platform, @oshun/core-team

Context and Problem Statement#

Isis has RunPod integration at adapter and handler layers, but key production behaviors remain incomplete:

  • cost and credit paths are stubbed
  • persistent provider job history is not implemented
  • typed handler modes are limited versus full Isis workflow catalog
  • endpoint lifecycle provisioning/drift control is incomplete

Current code evidence:

  • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts has placeholder implementations for model inventory, custom node inventory, cost estimation, credits, and job history.
  • docker/runpod/comfyui/handler.py validates only a limited typed mode set (txt2img, img2img, inpaint, upscale, workflow).
  • infra/terraform/cloudwatch/* includes RunPod monitoring, while endpoint provisioning automation is not defined as desired-state lifecycle resources.

Decision#

Adopt RunPod serverless as the primary asynchronous execution backend for Isis workflow jobs, governed by a capability map, cost ledger, and endpoint desired-state control plane.

Execution Model#

  • Use asynchronous RunPod job flow (run + status + cancel) for production jobs.
  • Reserve synchronous invocation for explicitly bounded low-latency tooling only.
  • Route jobs by workflowId via a versioned capability map to endpoint profiles.

Invocation Contract#

Provider payload must include:

json
{
  "workflowId": "string",
  "workflowVersion": "string",
  "input": {},
  "assets": {},
  "idempotencyKey": "string",
  "trace": {
    "jobId": "string",
    "projectId": "string",
    "userId": "string",
    "correlationId": "string"
  }
}

Cost and History Governance#

  • Persist per-job provider metadata including:
    • endpoint id/name
    • queued/start/completed timestamps
    • runtime seconds
    • pricing inputs
    • computed costUsd
  • Cost derivation must be deterministic and replayable from stored records.
  • Jobs missing billing fields are treated as policy failures.

Endpoint Strategy#

  • Define endpoint desired state in code (profile, GPU class, autoscaling limits, timeouts, image version, capability set).
  • Validate desired state in CI.
  • Run drift detection and reconciliation before releases.
  • Use staged rollout/canary by endpoint capability group.

Failure and Fallback Policy#

  • RunPod remains primary for serverless execution.
  • Secondary provider fallback is allowed only under explicit incident/failover policy and must be observable.
  • Unsupported workflow/category responses must be deterministic and machine-readable.

Decision Drivers#

  • True pay-per-second production accounting.
  • Full workflow catalog coverage with explicit capability routing.
  • Operational reliability with drift control and release safety.
  • Auditable history for debugging, billing, and compliance.

Considered Options#

Option A: Keep partial RunPod integration and manual ops (Rejected)#

Pros:

  • Minimal immediate engineering work.

Cons:

  • Incomplete billing/accountability.
  • High operational drift risk.
  • Weak incident diagnosis and release confidence.

Option B: Switch primary execution to RunComfy and defer RunPod (Rejected)#

Pros:

  • Lower immediate RunPod hardening effort.

Cons:

  • Conflicts with explicit serverless/pay-per-second objective.
  • Defers central remediation findings rather than closing them.

Option C: Harden RunPod as primary async backend with governance controls (Chosen)#

Pros:

  • Aligns directly to remediation objective.
  • Strong control over cost, capability, and reliability.
  • Enables deterministic rollout and operational ownership.

Cons:

  • Requires coordinated work across adapter, handler, infra, and observability.
  • Requires disciplined desired-state management and CI gates.

Consequences#

Positive#

  • RunPod usage becomes auditable, enforceable, and cost-accountable.
  • Workflow routing becomes explicit and testable.
  • Endpoint drift/regression risk is materially reduced.

Tradeoffs#

  • More upfront platform engineering for provisioning and reconciliation tooling.
  • More strict validation may initially surface hidden incompatibilities.

Implementation Rules#

  • Adapter methods for cost, credits, history, model/custom-node inventory must be non-placeholder in production profiles.
  • Handler must expose workflow-id capability contract in addition to legacy typed modes.
  • Endpoint capability map is versioned and release-gated.
  • Cost and history persistence must be queryable by job/project/user/workflow/endpoint.
  • Alerts required for cost spikes, queue growth, failure-rate regression, and missing billing fields.

Rollout Plan#

  1. Define and publish endpoint capability map and workflow routing policy.
  2. Replace placeholder adapter billing/history/inventory methods with real integrations.
  3. Extend handler contracts to catalog-driven workflow execution.
  4. Implement desired-state endpoint provisioning, drift detection, and reconcile.
  5. Add canary rollout policy and release gates tied to cost/SLO regressions.

Ticket Traceability#

  • WS00-005
  • F05-001 to F05-010
  • F06-001 to F06-010
  • F08-001 to F08-010
  • OBS-001 to OBS-010
  • REL-001 to REL-010

References#

  • Internal:
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts
    • docker/runpod/comfyui/handler.py
    • infra/terraform/cloudwatch/dashboards.tf
    • infra/terraform/cloudwatch/alarms.tf
    • docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md
  • External: