Eve library

GPU Jobs System

Guide

Revision 1

On this page

The GPU Jobs system provides a unified interface for dispatching compute-intensive tasks to GPU cloud providers. It handles job routing, cost estimation, provider selection, and monitoring across multiple providers including RunPod, Modal, AWS ECS, and local GPUs.

Table of Contents#

Overview#

Architecture#

text
┌─────────────────────────────────────────────────────────────┐
│                     GPU Jobs System                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   API       │  │  Dispatcher │  │   Cost Estimator    │  │
│  │  Routes     │──│             │──│                     │  │
│  └─────────────┘  └──────┬──────┘  └─────────────────────┘  │
│                          │                                   │
│  ┌───────────────────────┴───────────────────────┐          │
│  │              Provider Router                   │          │
│  └───────────────────────┬───────────────────────┘          │
│                          │                                   │
│  ┌──────────┬────────────┼────────────┬──────────┐          │
│  │ RunPod   │  Modal     │  AWS ECS   │  Local   │          │
│  │ Executor │  Executor  │  Executor  │  Executor│          │
│  └──────────┴────────────┴────────────┴──────────┘          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  State      │  │  Monitoring │  │   Warm Pool         │  │
│  │  Manager    │  │  & Alerts   │  │   Manager           │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Key Features#

  • Multi-Provider Support: Route jobs to RunPod, Modal, AWS ECS, or local GPUs
  • Intelligent Routing: Automatic provider selection based on cost, availability, and requirements
  • Cost Estimation: Accurate cost predictions before job submission
  • Budget Enforcement: Per-job and aggregate budget limits
  • Warm Pool Management: Pre-provisioned workers for reduced cold start
  • Comprehensive Monitoring: Job metrics, provider health, and alerting
  • Retry & Failover: Automatic retry with exponential backoff and provider failover

Quick Start#

Installation#

bash
pnpm add @yemaya/gpu-jobs

Basic Usage#

typescript
import {
  createJobDispatcher,
  createGPUCostEstimator,
  type GPUJobRequestBase,
} from '@yemaya/gpu-jobs';

// Create dispatcher
const costEstimator = createGPUCostEstimator();
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'runpod-serverless',
});

// Submit a job
const job: GPUJobRequestBase = {
  type: 'texture-upscale',
  priority: 'normal',
  gpuRequirements: {
    minVRAM: 8,
    preferredGPU: 'rtx-4090',
    canUseSpot: true,
    estimatedDuration: 60,
  },
  input: {
    textureUrl: 'https://storage.example.com/input.png',
    scaleFactor: 4,
    algorithm: 'real-esrgan-x4',
  },
  output: {
    destinationPath: 'textures/upscaled/',
    presignedUrls: true,
  },
};

// Get cost estimate first
const estimate = dispatcher.estimateCost(job);
console.log(`Estimated cost: $${estimate.estimatedCost.toFixed(4)}`);
console.log(`Estimated duration: ${estimate.estimatedDuration}s`);

// Submit the job
const result = await dispatcher.dispatch(job);
console.log(`Job ID: ${result.jobId}`);
console.log(`Provider: ${result.provider}`);

// Check status
const status = await dispatcher.getStatus(result.jobId);
console.log(`Status: ${status.status}`);
console.log(`Progress: ${status.progress.percent}%`);

Job Types#

The GPU Jobs system supports the following job types:

Job Type Description Typical GPU VRAM
texture-upscale AI texture upscaling (Real-ESRGAN, SwinIR) RTX 4090 8-12GB
gaussian-splatting 3D Gaussian Splatting training A100 24-80GB
nerf-training Neural Radiance Field training A100 24-80GB
blender-render Blender Cycles/EEVEE rendering RTX 4090 16-24GB
mesh-processing Mesh optimization, decimation, unwrapping RTX 4090 8-16GB
video-generation AI video generation (Runway, Sora) A100/H100 40-80GB
image-generation AI image generation (SDXL, Flux) RTX 4090 12-24GB
neural-bake Neural texture baking A10 16-24GB
ai-upscale General AI upscaling RTX 4090 8-16GB
voice-synthesis Text-to-speech synthesis L4 8-16GB
music-generation AI music generation A10 16-24GB
custom Custom workloads Any Varies

Job Request Structure#

typescript
interface GPUJobRequestBase {
  // Optional - auto-generated if not provided
  id?: GPUJobId;

  // Required
  type: GPUJobType;
  priority: 'urgent' | 'high' | 'normal' | 'low' | 'background';

  // GPU Requirements
  gpuRequirements: {
    minVRAM: number; // Minimum VRAM in GB
    preferredGPU?: GPUType; // Preferred GPU type
    gpuCount?: number; // Number of GPUs (default: 1)
    canUseSpot?: boolean; // Allow spot/preemptible instances
    estimatedDuration?: number; // Estimated duration in seconds
  };

  // Job-specific input parameters
  input: Record<string, unknown>;

  // Output configuration
  output: {
    destinationPath: string; // Where to store outputs
    presignedUrls?: boolean; // Generate presigned URLs
    urlExpiration?: number; // URL expiration in seconds
  };

  // Optional
  userId?: string;
  projectId?: string;
  organizationId?: string;
  webhookUrl?: string;
  timeout?: number; // Job timeout in seconds
  costConstraints?: {
    maxCost: number; // Maximum cost in USD
  };
}

Providers#

RunPod Serverless#

Best for: Short-duration jobs, variable workloads

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'runpod-serverless',
  providers: {
    'runpod-serverless': createRunPodServerlessExecutor({
      apiKey: process.env.RUNPOD_API_KEY,
      endpointId: process.env.RUNPOD_ENDPOINT_ID,
    }),
  },
});

Pricing: Pay per second, no minimum billing, ~3s cold start

RunPod Pod#

Best for: Long-running jobs, consistent workloads

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'runpod-pod',
  providers: {
    'runpod-pod': createRunPodPodExecutor({
      apiKey: process.env.RUNPOD_API_KEY,
      templateId: process.env.RUNPOD_TEMPLATE_ID,
    }),
  },
});

Pricing: Per-minute billing, 1-minute minimum, ~30s cold start

AWS ECS (Spot/On-Demand)#

Best for: Enterprise workloads, AWS ecosystem integration

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  providers: {
    'aws-ecs-spot': createAWSECSExecutor({
      region: 'us-east-1',
      cluster: 'gpu-jobs-cluster',
      taskDefinition: 'gpu-worker:latest',
      useSpot: true,
    }),
    'aws-ecs-ondemand': createAWSECSExecutor({
      region: 'us-east-1',
      cluster: 'gpu-jobs-cluster',
      taskDefinition: 'gpu-worker:latest',
      useSpot: false,
    }),
  },
});

Pricing: Spot ~70% cheaper but can be interrupted, On-demand guaranteed availability

Best for: Python-based workloads, rapid iteration

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'modal',
  providers: {
    modal: createModalExecutor({
      token: process.env.MODAL_TOKEN,
      appName: 'gpu-jobs',
    }),
  },
});

Pricing: Per-second billing, no cold start fee, excellent DX

Local#

Best for: Development, testing, on-premise GPUs

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'local',
  providers: {
    local: createLocalExecutor({
      gpuIds: [0, 1], // Available GPU indices
      maxConcurrent: 2,
    }),
  },
});

Pricing: Only electricity costs (~$0.10/GPU-hour estimate)

Cost Estimation#

Getting Cost Estimates#

typescript
// Single provider estimate
const estimate = costEstimator.estimateForProvider(
  jobRequest,
  'runpod-serverless',
  'rtx-4090'
);

console.log(`Provider: ${estimate.provider}`);
console.log(`GPU: ${estimate.gpuType}`);
console.log(`Estimated Cost: $${estimate.estimatedCost.toFixed(4)}`);
console.log(`Estimated Duration: ${estimate.estimatedDuration}s`);
console.log(`Breakdown:`, estimate.breakdown);

// All providers comparison
const allEstimates = costEstimator.estimateAllProviders(jobRequest);
allEstimates.forEach((est) => {
  console.log(
    `${est.provider}/${est.gpuType}: $${est.estimatedCost.toFixed(4)}`
  );
});

Cost Breakdown#

Each estimate includes a detailed breakdown:

typescript
interface CostBreakdown {
  compute: number; // GPU compute time
  transferIn: number; // Data download cost
  transferOut: number; // Data upload cost
  storage?: number; // Storage cost (if applicable)
  platformFee: number; // Provider platform fee
}

Budget Enforcement#

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  budgetConfig: {
    maxCostPerJob: 10.0, // Max $10 per job
    dailyBudget: 100.0, // Max $100 per day
    monthlyBudget: 1000.0, // Max $1000 per month
  },
});

// Jobs exceeding budget will be rejected
try {
  await dispatcher.dispatch(expensiveJob);
} catch (error) {
  if (error.code === 'BUDGET_EXCEEDED') {
    console.log('Job rejected: budget exceeded');
  }
}

API Reference#

See API Documentation for complete API reference.

Key Classes#

  • JobDispatcher - Main interface for submitting and managing jobs
  • GPUCostEstimator - Cost estimation across providers
  • GPUJobStateManager - Persistent job state management
  • WarmPoolManager - Pre-provisioned worker management
  • GPUJobsMonitoringService - Metrics and alerting

Key Functions#

  • createJobDispatcher(config) - Create a new dispatcher
  • createGPUCostEstimator(config) - Create a cost estimator
  • createWarmPoolManager(config) - Create warm pool manager
  • getGPUJobsMonitoringService() - Get monitoring singleton

Integration Guide#

With 3D Generation Package#

typescript
import { MeshProcessor } from '@yemaya/3d-generation';

// Enable GPU dispatch for heavy operations
const processor = new MeshProcessor({
  enableGPU: true,
  gpuConfig: {
    dispatcher: myDispatcher,
    minPolygonsForGPU: 100_000,
    maxCostPerOperation: 1.0,
  },
});

// Heavy operations automatically dispatch to GPU
const result = await processor.optimizeMesh(model, {
  targetPolyCount: 50_000,
  meshUrl: 'https://storage.example.com/model.glb',
});

With Video Package#

typescript
import { VideoGPUDispatchManager } from '@yemaya/video';

const videoDispatch = new VideoGPUDispatchManager({
  dispatcher: myDispatcher,
  maxCostPerJob: 5.0,
});

// Submit Blender render job
const jobId = await videoDispatch.submitBlenderRender({
  blendFileUrl: 'https://storage.example.com/scene.blend',
  outputPath: 'renders/output/',
  frameRange: { start: 1, end: 250 },
  resolution: { width: 1920, height: 1080 },
  engine: 'cycles',
  samples: 128,
  useGPU: true,
});

With Gaussian Splatting Package#

typescript
import { GPUTrainingBackend } from '@yemaya/gaussian-splatting';

const backend = new GPUTrainingBackend({
  dispatcher: myDispatcher,
  maxCostPerJob: 25.0,
});

// Training automatically dispatches to GPU cloud
await backend.initialize({
  pointCloudPath: '/data/colmap/points.ply',
  imageDir: '/data/images/',
  outputDir: '/output/splat/',
  config: { iterations: 30000 },
});

await backend.startTraining();

Monitoring#

Job Metrics#

typescript
import { getGPUJobsMonitoringService } from '@yemaya/gpu-jobs';

const monitoring = getGPUJobsMonitoringService();

// Get job metrics
const metrics = monitoring.getJobMetrics();
console.log(`Total Jobs: ${metrics.totalJobs}`);
console.log(`Completed: ${metrics.completedJobs}`);
console.log(`Failed: ${metrics.failedJobs}`);
console.log(`Success Rate: ${(metrics.successRate * 100).toFixed(1)}%`);
console.log(`Avg Duration: ${metrics.avgDuration}s`);
console.log(`Total Cost: $${metrics.totalCost.toFixed(2)}`);

Provider Health#

typescript
const providerMetrics = monitoring.getProviderMetrics();
for (const [provider, status] of Object.entries(providerMetrics)) {
  console.log(`${provider}:`);
  console.log(`  Status: ${status.health}`);
  console.log(`  Capacity: ${status.available}/${status.total}`);
  console.log(`  Queue Depth: ${status.queueDepth}`);
  console.log(`  Avg Latency: ${status.avgLatency}ms`);
}

Alerting#

typescript
const alerting = monitoring.getAlertManager();

// Configure alerts
alerting.addRule({
  id: 'high-failure-rate',
  condition: { metric: 'failure_rate', operator: '>', threshold: 0.1 },
  severity: 'critical',
  channels: ['slack', 'pagerduty'],
});

// Subscribe to alerts
alerting.on('alert-fired', (alert) => {
  console.log(`Alert: ${alert.rule.id} - ${alert.message}`);
});

Troubleshooting#

Common Issues#

Job stuck in "provisioning" state

  • Check provider API credentials
  • Verify GPU availability in the selected region
  • Check for rate limiting

Cost estimate significantly different from actual

  • Large file transfers can add unexpected costs
  • Cold start times vary by provider
  • Check for multi-GPU jobs

Job failed with "GPU out of memory"

  • Increase minVRAM requirement
  • Use a GPU with more memory (A100-80GB, H100)
  • Reduce batch size in job parameters

Provider unavailable

  • Check provider status page
  • Verify API key is valid
  • Check network connectivity

Debug Logging#

typescript
import { setLogLevel } from '@yemaya/gpu-jobs';

// Enable debug logging
setLogLevel('debug');

// Or for specific components
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');

Health Checks#

typescript
// Check system health
const health = await dispatcher.healthCheck();
console.log(`System Status: ${health.status}`);
console.log(`Providers:`, health.providers);
console.log(`Queue Depth: ${health.queueDepth}`);

Advanced Features#

Health Dashboard#

The health dashboard provides a comprehensive view of system status:

typescript
// Get full health dashboard
const health = await gpuJobsService.getHealthDashboard({
  includeAlerts: true,
  includeProviderDetails: true,
  includeComparison: true,
  includeSystemMetrics: true,
});

console.log('Status:', health.status); // 'healthy' | 'degraded' | 'unhealthy' | 'critical'
console.log('Active Alerts:', health.alerts);
console.log('Provider Comparison:', health.comparison);
console.log('Queue Metrics:', health.queue);
console.log('Budget Status:', health.budget);

API Endpoint:

text
GET /api/v1/gpu-jobs/health

A/B Testing#

Compare provider performance with A/B tests:

typescript
// Create A/B test
const test = await gpuJobsService.createABTest({
  name: 'RunPod vs Modal Performance',
  description: 'Compare latency and cost for texture upscaling',
  controlProvider: 'runpod-serverless',
  treatmentProvider: 'modal',
  trafficSplit: 50,
  targetSampleSize: 100,
  jobTypes: ['texture-upscale'],
});

// Get test results with statistical significance
const results = await gpuJobsService.getABTest(test.id);
console.log('Statistical Significance:', results.statisticalSignificance);
console.log('Recommended Winner:', results.winner);

API Endpoints:

text
POST   /api/v1/gpu-jobs/ab-tests           # Create A/B test
GET    /api/v1/gpu-jobs/ab-tests           # List A/B tests
GET    /api/v1/gpu-jobs/ab-tests/{testId}  # Get A/B test details
POST   /api/v1/gpu-jobs/ab-tests/{testId}/stop    # Stop A/B test
DELETE /api/v1/gpu-jobs/ab-tests/{testId}  # Delete A/B test

Circuit Breakers#

Automatic provider isolation with circuit breakers:

typescript
// Get circuit breaker status
const status =
  await gpuJobsService.getCircuitBreakerStatus('runpod-serverless');
console.log('State:', status.state); // 'closed' | 'open' | 'half-open'
console.log('Failure Count:', status.failureCount);
console.log('Success Rate:', status.successRate);

// Reset circuit breaker
await gpuJobsService.resetCircuitBreaker('runpod-serverless');

// Update configuration
await gpuJobsService.updateCircuitBreakerConfig('runpod-serverless', {
  failureThreshold: 5,
  resetTimeoutMs: 60000,
  halfOpenMaxRequests: 3,
});

API Endpoints:

text
GET  /api/v1/gpu-jobs/circuit-breakers                    # List all circuit breakers
GET  /api/v1/gpu-jobs/circuit-breakers/{provider}         # Get provider status
POST /api/v1/gpu-jobs/circuit-breakers/{provider}/reset   # Reset circuit breaker
PATCH /api/v1/gpu-jobs/circuit-breakers/{provider}/config # Update config

Dead Letter Queue (DLQ)#

Manage permanently failed jobs:

typescript
// List DLQ entries
const dlq = await gpuJobsService.listDLQ({
  failureReason: 'max_retries_exceeded',
  jobType: 'texture-upscale',
  limit: 20,
  sort: 'failedAt',
  order: 'desc',
});

// Get DLQ statistics
const stats = await gpuJobsService.getDLQStats();
console.log('Total Entries:', stats.totalEntries);
console.log('By Failure Reason:', stats.byFailureReason);
console.log('Total Cost Lost:', stats.totalCostLost);

// Replay a failed job
const result = await gpuJobsService.replayDLQEntry(entryId, {
  preferredProvider: 'modal', // Try different provider
  modifyParameters: { quality: 'high' },
});

// Bulk replay
const bulkResult = await gpuJobsService.replayDLQEntries(entryIds, {
  maxCost: 1.0,
});

// Purge old entries (with dry run)
const purgeResult = await gpuJobsService.purgeDLQ({
  olderThan: '2024-01-01',
  includeReplayed: true,
  dryRun: true, // Preview what would be purged
});

API Endpoints:

text
GET    /api/v1/gpu-jobs/dlq                    # List DLQ entries
GET    /api/v1/gpu-jobs/dlq/stats              # Get DLQ statistics
GET    /api/v1/gpu-jobs/dlq/{entryId}          # Get DLQ entry details
POST   /api/v1/gpu-jobs/dlq/{entryId}/replay   # Replay single entry
POST   /api/v1/gpu-jobs/dlq/replay             # Bulk replay entries
DELETE /api/v1/gpu-jobs/dlq/{entryId}          # Delete DLQ entry
POST   /api/v1/gpu-jobs/dlq/purge              # Purge DLQ entries

Routing Rules Configuration#

Configure job routing without code changes using JSON configuration:

typescript
import { RoutingRulesLoader, RoutingRulesEngine } from '@yemaya/gpu-jobs';

// Load from file
const config = RoutingRulesLoader.loadFromFile('./routing-config.json');
const engine = new RoutingRulesEngine(config);

// Evaluate rules for a job
const decision = engine.evaluateRules(jobRequest, {
  currentCapacity: providerCapacities,
  estimatedCosts: costEstimates,
});

console.log('Provider Order:', decision.providerOrder);
console.log('Matched Rules:', decision.matchedRules);
console.log('Applied Settings:', decision.appliedSettings);

See Routing Rules Guide for detailed configuration options.

Prometheus Metrics#

Export metrics in Prometheus format:

text
GET /api/v1/gpu-jobs/metrics

Available Metrics:

  • gpu_jobs_total - Total jobs by type, status, and provider
  • gpu_jobs_cost_total - Total cost incurred
  • gpu_jobs_duration_seconds - Job duration histogram
  • gpu_jobs_queue_depth - Current queue depth
  • gpu_jobs_provider_success_rate - Provider success rates
  • gpu_jobs_circuit_breaker_state - Circuit breaker states
  • gpu_jobs_dlq_entries - Dead letter queue size