# ADR-0003: RunPod Serverless Execution Strategy and Governance

**Status**: Accepted  
**Date**: 2026-02-21  
**Authors**: Isis ML Platform, Isis Runtime Engineering  
**Reviewers**: `@oshun/ml-team`, `@oshun/devops`, `@oshun/platform`, `@oshun/core-team`

## Context and Problem Statement

Isis has RunPod integration at adapter and handler layers, but key production behaviors remain incomplete:
- cost and credit paths are stubbed
- persistent provider job history is not implemented
- typed handler modes are limited versus full Isis workflow catalog
- endpoint lifecycle provisioning/drift control is incomplete

Current code evidence:
- `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts` has placeholder implementations for model inventory, custom node inventory, cost estimation, credits, and job history.
- `docker/runpod/comfyui/handler.py` validates only a limited typed mode set (`txt2img`, `img2img`, `inpaint`, `upscale`, `workflow`).
- `infra/terraform/cloudwatch/*` includes RunPod monitoring, while endpoint provisioning automation is not defined as desired-state lifecycle resources.

## Decision

Adopt **RunPod serverless as the primary asynchronous execution backend for Isis workflow jobs**, governed by a capability map, cost ledger, and endpoint desired-state control plane.

### Execution Model

- Use asynchronous RunPod job flow (`run` + `status` + `cancel`) for production jobs.
- Reserve synchronous invocation for explicitly bounded low-latency tooling only.
- Route jobs by `workflowId` via a versioned capability map to endpoint profiles.

### Invocation Contract

Provider payload must include:

```json
{
  "workflowId": "string",
  "workflowVersion": "string",
  "input": {},
  "assets": {},
  "idempotencyKey": "string",
  "trace": {
    "jobId": "string",
    "projectId": "string",
    "userId": "string",
    "correlationId": "string"
  }
}
```

### Cost and History Governance

- Persist per-job provider metadata including:
  - endpoint id/name
  - queued/start/completed timestamps
  - runtime seconds
  - pricing inputs
  - computed `costUsd`
- Cost derivation must be deterministic and replayable from stored records.
- Jobs missing billing fields are treated as policy failures.

### Endpoint Strategy

- Define endpoint desired state in code (profile, GPU class, autoscaling limits, timeouts, image version, capability set).
- Validate desired state in CI.
- Run drift detection and reconciliation before releases.
- Use staged rollout/canary by endpoint capability group.

### Failure and Fallback Policy

- RunPod remains primary for serverless execution.
- Secondary provider fallback is allowed only under explicit incident/failover policy and must be observable.
- Unsupported workflow/category responses must be deterministic and machine-readable.

## Decision Drivers

- True pay-per-second production accounting.
- Full workflow catalog coverage with explicit capability routing.
- Operational reliability with drift control and release safety.
- Auditable history for debugging, billing, and compliance.

## Considered Options

### Option A: Keep partial RunPod integration and manual ops (Rejected)

Pros:
- Minimal immediate engineering work.

Cons:
- Incomplete billing/accountability.
- High operational drift risk.
- Weak incident diagnosis and release confidence.

### Option B: Switch primary execution to RunComfy and defer RunPod (Rejected)

Pros:
- Lower immediate RunPod hardening effort.

Cons:
- Conflicts with explicit serverless/pay-per-second objective.
- Defers central remediation findings rather than closing them.

### Option C: Harden RunPod as primary async backend with governance controls (Chosen)

Pros:
- Aligns directly to remediation objective.
- Strong control over cost, capability, and reliability.
- Enables deterministic rollout and operational ownership.

Cons:
- Requires coordinated work across adapter, handler, infra, and observability.
- Requires disciplined desired-state management and CI gates.

## Consequences

### Positive

- RunPod usage becomes auditable, enforceable, and cost-accountable.
- Workflow routing becomes explicit and testable.
- Endpoint drift/regression risk is materially reduced.

### Tradeoffs

- More upfront platform engineering for provisioning and reconciliation tooling.
- More strict validation may initially surface hidden incompatibilities.

## Implementation Rules

- Adapter methods for cost, credits, history, model/custom-node inventory must be non-placeholder in production profiles.
- Handler must expose workflow-id capability contract in addition to legacy typed modes.
- Endpoint capability map is versioned and release-gated.
- Cost and history persistence must be queryable by job/project/user/workflow/endpoint.
- Alerts required for cost spikes, queue growth, failure-rate regression, and missing billing fields.

## Rollout Plan

1. Define and publish endpoint capability map and workflow routing policy.
2. Replace placeholder adapter billing/history/inventory methods with real integrations.
3. Extend handler contracts to catalog-driven workflow execution.
4. Implement desired-state endpoint provisioning, drift detection, and reconcile.
5. Add canary rollout policy and release gates tied to cost/SLO regressions.

## Ticket Traceability

- `WS00-005`
- `F05-001` to `F05-010`
- `F06-001` to `F06-010`
- `F08-001` to `F08-010`
- `OBS-001` to `OBS-010`
- `REL-001` to `REL-010`

## References

- Internal:
  - `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts`
  - `docker/runpod/comfyui/handler.py`
  - `infra/terraform/cloudwatch/dashboards.tf`
  - `infra/terraform/cloudwatch/alarms.tf`
  - `docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md`
- External:
  - [RunPod Serverless Endpoints](https://docs.runpod.io/serverless/endpoints/overview)
  - [RunPod API Operations](https://docs.runpod.io/serverless/endpoints/operations)
