The GPU Jobs system provides a unified interface for dispatching compute-intensive tasks to GPU cloud providers. It handles job routing, cost estimation, provider selection, and monitoring across multiple providers including RunPod, Modal, AWS ECS, and local GPUs.
Table of Contents#
- Overview
- Quick Start
- Job Types
- Providers
- Cost Estimation
- API Reference
- Integration Guide
- Monitoring
- Troubleshooting
Overview#
Architecture#
┌─────────────────────────────────────────────────────────────┐
│ GPU Jobs System │
├─────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ API │ │ Dispatcher │ │ Cost Estimator │ │
│ │ Routes │──│ │──│ │ │
│ └─────────────┘ └──────┬──────┘ └─────────────────────┘ │
│ │ │
│ ┌───────────────────────┴───────────────────────┐ │
│ │ Provider Router │ │
│ └───────────────────────┬───────────────────────┘ │
│ │ │
│ ┌──────────┬────────────┼────────────┬──────────┐ │
│ │ RunPod │ Modal │ AWS ECS │ Local │ │
│ │ Executor │ Executor │ Executor │ Executor│ │
│ └──────────┴────────────┴────────────┴──────────┘ │
├─────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ State │ │ Monitoring │ │ Warm Pool │ │
│ │ Manager │ │ & Alerts │ │ Manager │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Key Features#
- Multi-Provider Support: Route jobs to RunPod, Modal, AWS ECS, or local GPUs
- Intelligent Routing: Automatic provider selection based on cost, availability, and requirements
- Cost Estimation: Accurate cost predictions before job submission
- Budget Enforcement: Per-job and aggregate budget limits
- Warm Pool Management: Pre-provisioned workers for reduced cold start
- Comprehensive Monitoring: Job metrics, provider health, and alerting
- Retry & Failover: Automatic retry with exponential backoff and provider failover
Quick Start#
Installation#
pnpm add @yemaya/gpu-jobs
Basic Usage#
import {
createJobDispatcher,
createGPUCostEstimator,
type GPUJobRequestBase,
} from '@yemaya/gpu-jobs';
// Create dispatcher
const costEstimator = createGPUCostEstimator();
const dispatcher = createJobDispatcher({
costEstimator,
defaultProvider: 'runpod-serverless',
});
// Submit a job
const job: GPUJobRequestBase = {
type: 'texture-upscale',
priority: 'normal',
gpuRequirements: {
minVRAM: 8,
preferredGPU: 'rtx-4090',
canUseSpot: true,
estimatedDuration: 60,
},
input: {
textureUrl: 'https://storage.example.com/input.png',
scaleFactor: 4,
algorithm: 'real-esrgan-x4',
},
output: {
destinationPath: 'textures/upscaled/',
presignedUrls: true,
},
};
// Get cost estimate first
const estimate = dispatcher.estimateCost(job);
console.log(`Estimated cost: $${estimate.estimatedCost.toFixed(4)}`);
console.log(`Estimated duration: ${estimate.estimatedDuration}s`);
// Submit the job
const result = await dispatcher.dispatch(job);
console.log(`Job ID: ${result.jobId}`);
console.log(`Provider: ${result.provider}`);
// Check status
const status = await dispatcher.getStatus(result.jobId);
console.log(`Status: ${status.status}`);
console.log(`Progress: ${status.progress.percent}%`);
Job Types#
The GPU Jobs system supports the following job types:
| Job Type | Description | Typical GPU | VRAM |
|---|---|---|---|
texture-upscale |
AI texture upscaling (Real-ESRGAN, SwinIR) | RTX 4090 | 8-12GB |
gaussian-splatting |
3D Gaussian Splatting training | A100 | 24-80GB |
nerf-training |
Neural Radiance Field training | A100 | 24-80GB |
blender-render |
Blender Cycles/EEVEE rendering | RTX 4090 | 16-24GB |
mesh-processing |
Mesh optimization, decimation, unwrapping | RTX 4090 | 8-16GB |
video-generation |
AI video generation (Runway, Sora) | A100/H100 | 40-80GB |
image-generation |
AI image generation (SDXL, Flux) | RTX 4090 | 12-24GB |
neural-bake |
Neural texture baking | A10 | 16-24GB |
ai-upscale |
General AI upscaling | RTX 4090 | 8-16GB |
voice-synthesis |
Text-to-speech synthesis | L4 | 8-16GB |
music-generation |
AI music generation | A10 | 16-24GB |
custom |
Custom workloads | Any | Varies |
Job Request Structure#
interface GPUJobRequestBase {
// Optional - auto-generated if not provided
id?: GPUJobId;
// Required
type: GPUJobType;
priority: 'urgent' | 'high' | 'normal' | 'low' | 'background';
// GPU Requirements
gpuRequirements: {
minVRAM: number; // Minimum VRAM in GB
preferredGPU?: GPUType; // Preferred GPU type
gpuCount?: number; // Number of GPUs (default: 1)
canUseSpot?: boolean; // Allow spot/preemptible instances
estimatedDuration?: number; // Estimated duration in seconds
};
// Job-specific input parameters
input: Record<string, unknown>;
// Output configuration
output: {
destinationPath: string; // Where to store outputs
presignedUrls?: boolean; // Generate presigned URLs
urlExpiration?: number; // URL expiration in seconds
};
// Optional
userId?: string;
projectId?: string;
organizationId?: string;
webhookUrl?: string;
timeout?: number; // Job timeout in seconds
costConstraints?: {
maxCost: number; // Maximum cost in USD
};
}
Providers#
RunPod Serverless#
Best for: Short-duration jobs, variable workloads
const dispatcher = createJobDispatcher({
costEstimator,
defaultProvider: 'runpod-serverless',
providers: {
'runpod-serverless': createRunPodServerlessExecutor({
apiKey: process.env.RUNPOD_API_KEY,
endpointId: process.env.RUNPOD_ENDPOINT_ID,
}),
},
});
Pricing: Pay per second, no minimum billing, ~3s cold start
RunPod Pod#
Best for: Long-running jobs, consistent workloads
const dispatcher = createJobDispatcher({
costEstimator,
defaultProvider: 'runpod-pod',
providers: {
'runpod-pod': createRunPodPodExecutor({
apiKey: process.env.RUNPOD_API_KEY,
templateId: process.env.RUNPOD_TEMPLATE_ID,
}),
},
});
Pricing: Per-minute billing, 1-minute minimum, ~30s cold start
AWS ECS (Spot/On-Demand)#
Best for: Enterprise workloads, AWS ecosystem integration
const dispatcher = createJobDispatcher({
costEstimator,
providers: {
'aws-ecs-spot': createAWSECSExecutor({
region: 'us-east-1',
cluster: 'gpu-jobs-cluster',
taskDefinition: 'gpu-worker:latest',
useSpot: true,
}),
'aws-ecs-ondemand': createAWSECSExecutor({
region: 'us-east-1',
cluster: 'gpu-jobs-cluster',
taskDefinition: 'gpu-worker:latest',
useSpot: false,
}),
},
});
Pricing: Spot ~70% cheaper but can be interrupted, On-demand guaranteed availability
Modal#
Best for: Python-based workloads, rapid iteration
const dispatcher = createJobDispatcher({
costEstimator,
defaultProvider: 'modal',
providers: {
modal: createModalExecutor({
token: process.env.MODAL_TOKEN,
appName: 'gpu-jobs',
}),
},
});
Pricing: Per-second billing, no cold start fee, excellent DX
Local#
Best for: Development, testing, on-premise GPUs
const dispatcher = createJobDispatcher({
costEstimator,
defaultProvider: 'local',
providers: {
local: createLocalExecutor({
gpuIds: [0, 1], // Available GPU indices
maxConcurrent: 2,
}),
},
});
Pricing: Only electricity costs (~$0.10/GPU-hour estimate)
Cost Estimation#
Getting Cost Estimates#
// Single provider estimate
const estimate = costEstimator.estimateForProvider(
jobRequest,
'runpod-serverless',
'rtx-4090'
);
console.log(`Provider: ${estimate.provider}`);
console.log(`GPU: ${estimate.gpuType}`);
console.log(`Estimated Cost: $${estimate.estimatedCost.toFixed(4)}`);
console.log(`Estimated Duration: ${estimate.estimatedDuration}s`);
console.log(`Breakdown:`, estimate.breakdown);
// All providers comparison
const allEstimates = costEstimator.estimateAllProviders(jobRequest);
allEstimates.forEach((est) => {
console.log(
`${est.provider}/${est.gpuType}: $${est.estimatedCost.toFixed(4)}`
);
});
Cost Breakdown#
Each estimate includes a detailed breakdown:
interface CostBreakdown {
compute: number; // GPU compute time
transferIn: number; // Data download cost
transferOut: number; // Data upload cost
storage?: number; // Storage cost (if applicable)
platformFee: number; // Provider platform fee
}
Budget Enforcement#
const dispatcher = createJobDispatcher({
costEstimator,
budgetConfig: {
maxCostPerJob: 10.0, // Max $10 per job
dailyBudget: 100.0, // Max $100 per day
monthlyBudget: 1000.0, // Max $1000 per month
},
});
// Jobs exceeding budget will be rejected
try {
await dispatcher.dispatch(expensiveJob);
} catch (error) {
if (error.code === 'BUDGET_EXCEEDED') {
console.log('Job rejected: budget exceeded');
}
}
API Reference#
See API Documentation for complete API reference.
Key Classes#
JobDispatcher- Main interface for submitting and managing jobsGPUCostEstimator- Cost estimation across providersGPUJobStateManager- Persistent job state managementWarmPoolManager- Pre-provisioned worker managementGPUJobsMonitoringService- Metrics and alerting
Key Functions#
createJobDispatcher(config)- Create a new dispatchercreateGPUCostEstimator(config)- Create a cost estimatorcreateWarmPoolManager(config)- Create warm pool managergetGPUJobsMonitoringService()- Get monitoring singleton
Integration Guide#
With 3D Generation Package#
import { MeshProcessor } from '@yemaya/3d-generation';
// Enable GPU dispatch for heavy operations
const processor = new MeshProcessor({
enableGPU: true,
gpuConfig: {
dispatcher: myDispatcher,
minPolygonsForGPU: 100_000,
maxCostPerOperation: 1.0,
},
});
// Heavy operations automatically dispatch to GPU
const result = await processor.optimizeMesh(model, {
targetPolyCount: 50_000,
meshUrl: 'https://storage.example.com/model.glb',
});
With Video Package#
import { VideoGPUDispatchManager } from '@yemaya/video';
const videoDispatch = new VideoGPUDispatchManager({
dispatcher: myDispatcher,
maxCostPerJob: 5.0,
});
// Submit Blender render job
const jobId = await videoDispatch.submitBlenderRender({
blendFileUrl: 'https://storage.example.com/scene.blend',
outputPath: 'renders/output/',
frameRange: { start: 1, end: 250 },
resolution: { width: 1920, height: 1080 },
engine: 'cycles',
samples: 128,
useGPU: true,
});
With Gaussian Splatting Package#
import { GPUTrainingBackend } from '@yemaya/gaussian-splatting';
const backend = new GPUTrainingBackend({
dispatcher: myDispatcher,
maxCostPerJob: 25.0,
});
// Training automatically dispatches to GPU cloud
await backend.initialize({
pointCloudPath: '/data/colmap/points.ply',
imageDir: '/data/images/',
outputDir: '/output/splat/',
config: { iterations: 30000 },
});
await backend.startTraining();
Monitoring#
Job Metrics#
import { getGPUJobsMonitoringService } from '@yemaya/gpu-jobs';
const monitoring = getGPUJobsMonitoringService();
// Get job metrics
const metrics = monitoring.getJobMetrics();
console.log(`Total Jobs: ${metrics.totalJobs}`);
console.log(`Completed: ${metrics.completedJobs}`);
console.log(`Failed: ${metrics.failedJobs}`);
console.log(`Success Rate: ${(metrics.successRate * 100).toFixed(1)}%`);
console.log(`Avg Duration: ${metrics.avgDuration}s`);
console.log(`Total Cost: $${metrics.totalCost.toFixed(2)}`);
Provider Health#
const providerMetrics = monitoring.getProviderMetrics();
for (const [provider, status] of Object.entries(providerMetrics)) {
console.log(`${provider}:`);
console.log(` Status: ${status.health}`);
console.log(` Capacity: ${status.available}/${status.total}`);
console.log(` Queue Depth: ${status.queueDepth}`);
console.log(` Avg Latency: ${status.avgLatency}ms`);
}
Alerting#
const alerting = monitoring.getAlertManager();
// Configure alerts
alerting.addRule({
id: 'high-failure-rate',
condition: { metric: 'failure_rate', operator: '>', threshold: 0.1 },
severity: 'critical',
channels: ['slack', 'pagerduty'],
});
// Subscribe to alerts
alerting.on('alert-fired', (alert) => {
console.log(`Alert: ${alert.rule.id} - ${alert.message}`);
});
Troubleshooting#
Common Issues#
Job stuck in "provisioning" state
- Check provider API credentials
- Verify GPU availability in the selected region
- Check for rate limiting
Cost estimate significantly different from actual
- Large file transfers can add unexpected costs
- Cold start times vary by provider
- Check for multi-GPU jobs
Job failed with "GPU out of memory"
- Increase
minVRAMrequirement - Use a GPU with more memory (A100-80GB, H100)
- Reduce batch size in job parameters
Provider unavailable
- Check provider status page
- Verify API key is valid
- Check network connectivity
Debug Logging#
import { setLogLevel } from '@yemaya/gpu-jobs';
// Enable debug logging
setLogLevel('debug');
// Or for specific components
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');
Health Checks#
// Check system health
const health = await dispatcher.healthCheck();
console.log(`System Status: ${health.status}`);
console.log(`Providers:`, health.providers);
console.log(`Queue Depth: ${health.queueDepth}`);
Advanced Features#
Health Dashboard#
The health dashboard provides a comprehensive view of system status:
// Get full health dashboard
const health = await gpuJobsService.getHealthDashboard({
includeAlerts: true,
includeProviderDetails: true,
includeComparison: true,
includeSystemMetrics: true,
});
console.log('Status:', health.status); // 'healthy' | 'degraded' | 'unhealthy' | 'critical'
console.log('Active Alerts:', health.alerts);
console.log('Provider Comparison:', health.comparison);
console.log('Queue Metrics:', health.queue);
console.log('Budget Status:', health.budget);
API Endpoint:
GET /api/v1/gpu-jobs/health
A/B Testing#
Compare provider performance with A/B tests:
// Create A/B test
const test = await gpuJobsService.createABTest({
name: 'RunPod vs Modal Performance',
description: 'Compare latency and cost for texture upscaling',
controlProvider: 'runpod-serverless',
treatmentProvider: 'modal',
trafficSplit: 50,
targetSampleSize: 100,
jobTypes: ['texture-upscale'],
});
// Get test results with statistical significance
const results = await gpuJobsService.getABTest(test.id);
console.log('Statistical Significance:', results.statisticalSignificance);
console.log('Recommended Winner:', results.winner);
API Endpoints:
POST /api/v1/gpu-jobs/ab-tests # Create A/B test
GET /api/v1/gpu-jobs/ab-tests # List A/B tests
GET /api/v1/gpu-jobs/ab-tests/{testId} # Get A/B test details
POST /api/v1/gpu-jobs/ab-tests/{testId}/stop # Stop A/B test
DELETE /api/v1/gpu-jobs/ab-tests/{testId} # Delete A/B test
Circuit Breakers#
Automatic provider isolation with circuit breakers:
// Get circuit breaker status
const status =
await gpuJobsService.getCircuitBreakerStatus('runpod-serverless');
console.log('State:', status.state); // 'closed' | 'open' | 'half-open'
console.log('Failure Count:', status.failureCount);
console.log('Success Rate:', status.successRate);
// Reset circuit breaker
await gpuJobsService.resetCircuitBreaker('runpod-serverless');
// Update configuration
await gpuJobsService.updateCircuitBreakerConfig('runpod-serverless', {
failureThreshold: 5,
resetTimeoutMs: 60000,
halfOpenMaxRequests: 3,
});
API Endpoints:
GET /api/v1/gpu-jobs/circuit-breakers # List all circuit breakers
GET /api/v1/gpu-jobs/circuit-breakers/{provider} # Get provider status
POST /api/v1/gpu-jobs/circuit-breakers/{provider}/reset # Reset circuit breaker
PATCH /api/v1/gpu-jobs/circuit-breakers/{provider}/config # Update config
Dead Letter Queue (DLQ)#
Manage permanently failed jobs:
// List DLQ entries
const dlq = await gpuJobsService.listDLQ({
failureReason: 'max_retries_exceeded',
jobType: 'texture-upscale',
limit: 20,
sort: 'failedAt',
order: 'desc',
});
// Get DLQ statistics
const stats = await gpuJobsService.getDLQStats();
console.log('Total Entries:', stats.totalEntries);
console.log('By Failure Reason:', stats.byFailureReason);
console.log('Total Cost Lost:', stats.totalCostLost);
// Replay a failed job
const result = await gpuJobsService.replayDLQEntry(entryId, {
preferredProvider: 'modal', // Try different provider
modifyParameters: { quality: 'high' },
});
// Bulk replay
const bulkResult = await gpuJobsService.replayDLQEntries(entryIds, {
maxCost: 1.0,
});
// Purge old entries (with dry run)
const purgeResult = await gpuJobsService.purgeDLQ({
olderThan: '2024-01-01',
includeReplayed: true,
dryRun: true, // Preview what would be purged
});
API Endpoints:
GET /api/v1/gpu-jobs/dlq # List DLQ entries
GET /api/v1/gpu-jobs/dlq/stats # Get DLQ statistics
GET /api/v1/gpu-jobs/dlq/{entryId} # Get DLQ entry details
POST /api/v1/gpu-jobs/dlq/{entryId}/replay # Replay single entry
POST /api/v1/gpu-jobs/dlq/replay # Bulk replay entries
DELETE /api/v1/gpu-jobs/dlq/{entryId} # Delete DLQ entry
POST /api/v1/gpu-jobs/dlq/purge # Purge DLQ entries
Routing Rules Configuration#
Configure job routing without code changes using JSON configuration:
import { RoutingRulesLoader, RoutingRulesEngine } from '@yemaya/gpu-jobs';
// Load from file
const config = RoutingRulesLoader.loadFromFile('./routing-config.json');
const engine = new RoutingRulesEngine(config);
// Evaluate rules for a job
const decision = engine.evaluateRules(jobRequest, {
currentCapacity: providerCapacities,
estimatedCosts: costEstimates,
});
console.log('Provider Order:', decision.providerOrder);
console.log('Matched Rules:', decision.matchedRules);
console.log('Applied Settings:', decision.appliedSettings);
See Routing Rules Guide for detailed configuration options.
Prometheus Metrics#
Export metrics in Prometheus format:
GET /api/v1/gpu-jobs/metrics
Available Metrics:
gpu_jobs_total- Total jobs by type, status, and providergpu_jobs_cost_total- Total cost incurredgpu_jobs_duration_seconds- Job duration histogramgpu_jobs_queue_depth- Current queue depthgpu_jobs_provider_success_rate- Provider success ratesgpu_jobs_circuit_breaker_state- Circuit breaker statesgpu_jobs_dlq_entries- Dead letter queue size