# ISIS Cost Backfill Design (MIG-004)

Date: 2026-02-22  
Checklist Item: `MIG-004`  
Ticket: `ISIS-0173`

## Goal

Backfill missing historical cost metadata for terminal jobs using provider log
reconciliation plus deterministic runtime estimation fallbacks.

## Scope

Eligible records are terminal jobs in `jobs`:

- `completed`
- `failed`
- `cancelled`

A record is considered missing cost data when no non-negative canonical cost is
present in existing job/output metadata aliases (for example
`metadata.estimatedCostUsd` or `output.metadata.estimatedCostUsd`).

## Backfill Strategy

For each eligible job, the backfill pipeline performs:

1. Provider log reconciliation:
   - Match by `jobId` first, then `queueJobId`.
   - Prefer explicit `estimatedCostUsd` from provider logs when available.
   - Otherwise derive cost from log `executionTimeMs` and provider pricing
     configuration.
2. Metadata reconciliation:
   - Accept existing cost aliases when present.
3. Runtime estimation fallback:
   - Use runtime metadata fields (`executionTimeMs`, worker metrics duration,
     terminal timestamps) plus configured provider rates.

When a value is resolved, backfill writes:

- `metadata.estimatedCostUsd`
- `metadata.costBackfill` audit block (`source`, `backfilledAt`, optional
  provider/runtime/endpoint details)
- `output.metadata.estimatedCostUsd` (and missing runtime/provider fields when
  available)

## Source Resolution Rules

Deterministic precedence:

1. `provider_logs.estimated_cost_usd`
2. `provider_logs.execution_time_ms`
3. Explicit metadata aliases (`output.metadata.*`, `job.metadata.*`)
4. Runtime-driven estimation from metadata and timestamps

## Operational Interfaces

### API (admin)

- `POST /api/v1/jobs/admin/costs/backfill`

Request body:

```json
{
  "limit": 2000,
  "dryRun": false,
  "providerLogs": []
}
```

Access requirements:

- Bearer auth role: `admin`, `ops`, or `sre`
- API key permission: `jobs:queue:admin`

### CLI

- `isis jobs backfill-costs --dry-run --provider-log-file ./runpod-logs.json`

## Result Contract

Backfill response includes:

- `scanned`
- `missingCostData`
- `backfilled`
- `unresolved`
- `failed`
- `providerLogsProcessed`
- `providerLogsMatched`
- `backfilledJobIds`
- `unresolvedJobIds`
- `failedJobIds`
- `sourceCounts`
- `providerCounts`
- `checkedAt`

## Safety Controls

- Dry-run mode for impact estimation without writes.
- Bounded scan with hard max limit (`50000`) and page batching.
- Provider logs are optional and validated for identifier + cost/runtime signal.
- Ambiguous or insufficient data remains unresolved for operator review.
