Status: Accepted
Date: 2026-02-21
Authors: Isis Platform, Isis API, Oshun Platform Engineering
Reviewers: @oshun/core-team, @oshun/isis-team, @oshun/platform, @oshun/devops
Context and Problem Statement#
Isis currently persists submitted jobs directly to PostgreSQL and marks them pending at submit time. Queue state is inferred from database rows and rough wait-time estimation, but no dedicated producer/consumer queue exists in the generation API path.
Current code evidence:
apps/isis/generation-api/src/services/job.service.tswrites jobs directly and computes queue position fromstatus = 'pending'.apps/isis/generation-api/src/app.tsreadiness includes placeholder checks (queue: 'ok',storage: 'ok') instead of dependency-truthful queue health.libs/isis/client/src/resources/generation.tsexposes queue-oriented operations that require a real queue backend contract.
This creates an execution-critical risk: jobs can be accepted without deterministic execution guarantees.
Decision#
Adopt Redis + BullMQ as the canonical execution queue for Isis generation jobs, with a transactional outbox pattern and a shared, versioned job envelope contract.
Normative Queue Topology#
isis:jobs:submit(primary queue)isis:jobs:dlq(dead-letter queue)isis:jobs:events(lifecycle event stream/topic)isis:jobs:outbox(database-backed outbox relay source)
Canonical Job Envelope v1#
Every enqueued message must follow this minimum shape:
{
"jobId": "uuid",
"type": "text-to-image",
"input": {},
"priority": "normal",
"metadata": {},
"idempotencyKey": "string"
}
Required semantics:
jobId: immutable primary correlation key.idempotencyKey: required for dedupe at producer and consumer boundaries.metadata: must carryuserId,projectId, andcorrelationIdwhen available.
State Machine#
pending is removed from production submission flow and replaced with:
queued -> running -> completed | failed | cancelled
pending remains only for legacy migration/backfill compatibility.
Decision Drivers#
- Deterministic execution from API submit to worker consumption.
- Safe retry and idempotency guarantees.
- Clear cancellation semantics.
- Operational visibility (depth, lag, throughput, failure classes).
- Compatibility with existing TypeScript/Node services and Redis footprint.
Considered Options#
Option A: Continue DB-only queue emulation (Rejected)#
Pros:
- Minimal short-term code change.
Cons:
- No durable producer/consumer execution contract.
- No native retry/DLQ primitives.
- Queue position and latency are estimations, not queue truth.
Option B: In-process memory queue per service (Rejected)#
Pros:
- Fast local implementation.
Cons:
- Not horizontally safe.
- Loses in-flight work on process restart.
- Cannot support controlled replays or DLQ operations.
Option C: BullMQ + Redis + Outbox (Chosen)#
Pros:
- Production-grade queue semantics (retry, backoff, priorities, delayed jobs).
- Strong operational tooling and clear queue introspection.
- Natural fit for Node/TypeScript and existing platform patterns.
Cons:
- Requires Redis dependency hardening and runtime configuration.
- Requires disciplined outbox and replay operations.
Consequences#
Positive#
- Submit path and execution path are explicitly connected and auditable.
- Retry/DLQ behavior is standardized.
- Queue position and throughput come from runtime queue state.
- Production readiness checks become dependency-truthful.
Tradeoffs#
- Additional operational dependency (Redis/BullMQ) and alerting surface.
- Requires migration of legacy
pendingjobs and backfill tooling.
Implementation Rules#
- Producer writes
jobs+outboxin a single transaction. - Outbox relay is the only process that enqueues to BullMQ.
- Consumers must ACK only after durable status write.
- Retry policy: capped attempts + exponential backoff.
- DLQ replay requires explicit operator action with audit log.
- Queue configuration env vars are required in production startup.
Rollout Plan#
- Introduce shared envelope schema package and producer module.
- Add outbox table/relay and idempotency constraints.
- Switch submit path from DB-only to outbox+queue.
- Wire worker status callbacks for deterministic lifecycle updates.
- Enable queue health/readiness checks and operational runbook.
Ticket Traceability#
WS00-005F01-001toF01-015F11-001toF11-010OBS-001toOBS-010
References#
- Internal:
apps/isis/generation-api/src/services/job.service.tsapps/isis/generation-api/src/app.tsdocs/domains/isis/isis-domain-completeness-audit-2026-02-21.md
- External: