# GPU Jobs System

The GPU Jobs system provides a unified interface for dispatching
compute-intensive tasks to GPU cloud providers. It handles job routing, cost
estimation, provider selection, and monitoring across multiple providers
including RunPod, Modal, AWS ECS, and local GPUs.

## Table of Contents

- [Overview](#overview)
- [Quick Start](#quick-start)
- [Job Types](#job-types)
- [Providers](#providers)
- [Cost Estimation](#cost-estimation)
- [API Reference](#api-reference)
- [Integration Guide](#integration-guide)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)

## Overview

### Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     GPU Jobs System                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   API       │  │  Dispatcher │  │   Cost Estimator    │  │
│  │  Routes     │──│             │──│                     │  │
│  └─────────────┘  └──────┬──────┘  └─────────────────────┘  │
│                          │                                   │
│  ┌───────────────────────┴───────────────────────┐          │
│  │              Provider Router                   │          │
│  └───────────────────────┬───────────────────────┘          │
│                          │                                   │
│  ┌──────────┬────────────┼────────────┬──────────┐          │
│  │ RunPod   │  Modal     │  AWS ECS   │  Local   │          │
│  │ Executor │  Executor  │  Executor  │  Executor│          │
│  └──────────┴────────────┴────────────┴──────────┘          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  State      │  │  Monitoring │  │   Warm Pool         │  │
│  │  Manager    │  │  & Alerts   │  │   Manager           │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

### Key Features

- **Multi-Provider Support**: Route jobs to RunPod, Modal, AWS ECS, or local
  GPUs
- **Intelligent Routing**: Automatic provider selection based on cost,
  availability, and requirements
- **Cost Estimation**: Accurate cost predictions before job submission
- **Budget Enforcement**: Per-job and aggregate budget limits
- **Warm Pool Management**: Pre-provisioned workers for reduced cold start
- **Comprehensive Monitoring**: Job metrics, provider health, and alerting
- **Retry & Failover**: Automatic retry with exponential backoff and provider
  failover

## Quick Start

### Installation

```bash
pnpm add @yemaya/gpu-jobs
```

### Basic Usage

```typescript
import {
  createJobDispatcher,
  createGPUCostEstimator,
  type GPUJobRequestBase,
} from '@yemaya/gpu-jobs';

// Create dispatcher
const costEstimator = createGPUCostEstimator();
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'runpod-serverless',
});

// Submit a job
const job: GPUJobRequestBase = {
  type: 'texture-upscale',
  priority: 'normal',
  gpuRequirements: {
    minVRAM: 8,
    preferredGPU: 'rtx-4090',
    canUseSpot: true,
    estimatedDuration: 60,
  },
  input: {
    textureUrl: 'https://storage.example.com/input.png',
    scaleFactor: 4,
    algorithm: 'real-esrgan-x4',
  },
  output: {
    destinationPath: 'textures/upscaled/',
    presignedUrls: true,
  },
};

// Get cost estimate first
const estimate = dispatcher.estimateCost(job);
console.log(`Estimated cost: $${estimate.estimatedCost.toFixed(4)}`);
console.log(`Estimated duration: ${estimate.estimatedDuration}s`);

// Submit the job
const result = await dispatcher.dispatch(job);
console.log(`Job ID: ${result.jobId}`);
console.log(`Provider: ${result.provider}`);

// Check status
const status = await dispatcher.getStatus(result.jobId);
console.log(`Status: ${status.status}`);
console.log(`Progress: ${status.progress.percent}%`);
```

## Job Types

The GPU Jobs system supports the following job types:

| Job Type             | Description                                | Typical GPU | VRAM    |
| -------------------- | ------------------------------------------ | ----------- | ------- |
| `texture-upscale`    | AI texture upscaling (Real-ESRGAN, SwinIR) | RTX 4090    | 8-12GB  |
| `gaussian-splatting` | 3D Gaussian Splatting training             | A100        | 24-80GB |
| `nerf-training`      | Neural Radiance Field training             | A100        | 24-80GB |
| `blender-render`     | Blender Cycles/EEVEE rendering             | RTX 4090    | 16-24GB |
| `mesh-processing`    | Mesh optimization, decimation, unwrapping  | RTX 4090    | 8-16GB  |
| `video-generation`   | AI video generation (Runway, Sora)         | A100/H100   | 40-80GB |
| `image-generation`   | AI image generation (SDXL, Flux)           | RTX 4090    | 12-24GB |
| `neural-bake`        | Neural texture baking                      | A10         | 16-24GB |
| `ai-upscale`         | General AI upscaling                       | RTX 4090    | 8-16GB  |
| `voice-synthesis`    | Text-to-speech synthesis                   | L4          | 8-16GB  |
| `music-generation`   | AI music generation                        | A10         | 16-24GB |
| `custom`             | Custom workloads                           | Any         | Varies  |

### Job Request Structure

```typescript
interface GPUJobRequestBase {
  // Optional - auto-generated if not provided
  id?: GPUJobId;

  // Required
  type: GPUJobType;
  priority: 'urgent' | 'high' | 'normal' | 'low' | 'background';

  // GPU Requirements
  gpuRequirements: {
    minVRAM: number; // Minimum VRAM in GB
    preferredGPU?: GPUType; // Preferred GPU type
    gpuCount?: number; // Number of GPUs (default: 1)
    canUseSpot?: boolean; // Allow spot/preemptible instances
    estimatedDuration?: number; // Estimated duration in seconds
  };

  // Job-specific input parameters
  input: Record<string, unknown>;

  // Output configuration
  output: {
    destinationPath: string; // Where to store outputs
    presignedUrls?: boolean; // Generate presigned URLs
    urlExpiration?: number; // URL expiration in seconds
  };

  // Optional
  userId?: string;
  projectId?: string;
  organizationId?: string;
  webhookUrl?: string;
  timeout?: number; // Job timeout in seconds
  costConstraints?: {
    maxCost: number; // Maximum cost in USD
  };
}
```

## Providers

### RunPod Serverless

Best for: Short-duration jobs, variable workloads

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'runpod-serverless',
  providers: {
    'runpod-serverless': createRunPodServerlessExecutor({
      apiKey: process.env.RUNPOD_API_KEY,
      endpointId: process.env.RUNPOD_ENDPOINT_ID,
    }),
  },
});
```

**Pricing**: Pay per second, no minimum billing, ~3s cold start

### RunPod Pod

Best for: Long-running jobs, consistent workloads

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'runpod-pod',
  providers: {
    'runpod-pod': createRunPodPodExecutor({
      apiKey: process.env.RUNPOD_API_KEY,
      templateId: process.env.RUNPOD_TEMPLATE_ID,
    }),
  },
});
```

**Pricing**: Per-minute billing, 1-minute minimum, ~30s cold start

### AWS ECS (Spot/On-Demand)

Best for: Enterprise workloads, AWS ecosystem integration

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  providers: {
    'aws-ecs-spot': createAWSECSExecutor({
      region: 'us-east-1',
      cluster: 'gpu-jobs-cluster',
      taskDefinition: 'gpu-worker:latest',
      useSpot: true,
    }),
    'aws-ecs-ondemand': createAWSECSExecutor({
      region: 'us-east-1',
      cluster: 'gpu-jobs-cluster',
      taskDefinition: 'gpu-worker:latest',
      useSpot: false,
    }),
  },
});
```

**Pricing**: Spot ~70% cheaper but can be interrupted, On-demand guaranteed
availability

### Modal

Best for: Python-based workloads, rapid iteration

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'modal',
  providers: {
    modal: createModalExecutor({
      token: process.env.MODAL_TOKEN,
      appName: 'gpu-jobs',
    }),
  },
});
```

**Pricing**: Per-second billing, no cold start fee, excellent DX

### Local

Best for: Development, testing, on-premise GPUs

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  defaultProvider: 'local',
  providers: {
    local: createLocalExecutor({
      gpuIds: [0, 1], // Available GPU indices
      maxConcurrent: 2,
    }),
  },
});
```

**Pricing**: Only electricity costs (~$0.10/GPU-hour estimate)

## Cost Estimation

### Getting Cost Estimates

```typescript
// Single provider estimate
const estimate = costEstimator.estimateForProvider(
  jobRequest,
  'runpod-serverless',
  'rtx-4090'
);

console.log(`Provider: ${estimate.provider}`);
console.log(`GPU: ${estimate.gpuType}`);
console.log(`Estimated Cost: $${estimate.estimatedCost.toFixed(4)}`);
console.log(`Estimated Duration: ${estimate.estimatedDuration}s`);
console.log(`Breakdown:`, estimate.breakdown);

// All providers comparison
const allEstimates = costEstimator.estimateAllProviders(jobRequest);
allEstimates.forEach((est) => {
  console.log(
    `${est.provider}/${est.gpuType}: $${est.estimatedCost.toFixed(4)}`
  );
});
```

### Cost Breakdown

Each estimate includes a detailed breakdown:

```typescript
interface CostBreakdown {
  compute: number; // GPU compute time
  transferIn: number; // Data download cost
  transferOut: number; // Data upload cost
  storage?: number; // Storage cost (if applicable)
  platformFee: number; // Provider platform fee
}
```

### Budget Enforcement

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  budgetConfig: {
    maxCostPerJob: 10.0, // Max $10 per job
    dailyBudget: 100.0, // Max $100 per day
    monthlyBudget: 1000.0, // Max $1000 per month
  },
});

// Jobs exceeding budget will be rejected
try {
  await dispatcher.dispatch(expensiveJob);
} catch (error) {
  if (error.code === 'BUDGET_EXCEEDED') {
    console.log('Job rejected: budget exceeded');
  }
}
```

## API Reference

See [API Documentation](./api.md) for complete API reference.

### Key Classes

- `JobDispatcher` - Main interface for submitting and managing jobs
- `GPUCostEstimator` - Cost estimation across providers
- `GPUJobStateManager` - Persistent job state management
- `WarmPoolManager` - Pre-provisioned worker management
- `GPUJobsMonitoringService` - Metrics and alerting

### Key Functions

- `createJobDispatcher(config)` - Create a new dispatcher
- `createGPUCostEstimator(config)` - Create a cost estimator
- `createWarmPoolManager(config)` - Create warm pool manager
- `getGPUJobsMonitoringService()` - Get monitoring singleton

## Integration Guide

### With 3D Generation Package

```typescript
import { MeshProcessor } from '@yemaya/3d-generation';

// Enable GPU dispatch for heavy operations
const processor = new MeshProcessor({
  enableGPU: true,
  gpuConfig: {
    dispatcher: myDispatcher,
    minPolygonsForGPU: 100_000,
    maxCostPerOperation: 1.0,
  },
});

// Heavy operations automatically dispatch to GPU
const result = await processor.optimizeMesh(model, {
  targetPolyCount: 50_000,
  meshUrl: 'https://storage.example.com/model.glb',
});
```

### With Video Package

```typescript
import { VideoGPUDispatchManager } from '@yemaya/video';

const videoDispatch = new VideoGPUDispatchManager({
  dispatcher: myDispatcher,
  maxCostPerJob: 5.0,
});

// Submit Blender render job
const jobId = await videoDispatch.submitBlenderRender({
  blendFileUrl: 'https://storage.example.com/scene.blend',
  outputPath: 'renders/output/',
  frameRange: { start: 1, end: 250 },
  resolution: { width: 1920, height: 1080 },
  engine: 'cycles',
  samples: 128,
  useGPU: true,
});
```

### With Gaussian Splatting Package

```typescript
import { GPUTrainingBackend } from '@yemaya/gaussian-splatting';

const backend = new GPUTrainingBackend({
  dispatcher: myDispatcher,
  maxCostPerJob: 25.0,
});

// Training automatically dispatches to GPU cloud
await backend.initialize({
  pointCloudPath: '/data/colmap/points.ply',
  imageDir: '/data/images/',
  outputDir: '/output/splat/',
  config: { iterations: 30000 },
});

await backend.startTraining();
```

## Monitoring

### Job Metrics

```typescript
import { getGPUJobsMonitoringService } from '@yemaya/gpu-jobs';

const monitoring = getGPUJobsMonitoringService();

// Get job metrics
const metrics = monitoring.getJobMetrics();
console.log(`Total Jobs: ${metrics.totalJobs}`);
console.log(`Completed: ${metrics.completedJobs}`);
console.log(`Failed: ${metrics.failedJobs}`);
console.log(`Success Rate: ${(metrics.successRate * 100).toFixed(1)}%`);
console.log(`Avg Duration: ${metrics.avgDuration}s`);
console.log(`Total Cost: $${metrics.totalCost.toFixed(2)}`);
```

### Provider Health

```typescript
const providerMetrics = monitoring.getProviderMetrics();
for (const [provider, status] of Object.entries(providerMetrics)) {
  console.log(`${provider}:`);
  console.log(`  Status: ${status.health}`);
  console.log(`  Capacity: ${status.available}/${status.total}`);
  console.log(`  Queue Depth: ${status.queueDepth}`);
  console.log(`  Avg Latency: ${status.avgLatency}ms`);
}
```

### Alerting

```typescript
const alerting = monitoring.getAlertManager();

// Configure alerts
alerting.addRule({
  id: 'high-failure-rate',
  condition: { metric: 'failure_rate', operator: '>', threshold: 0.1 },
  severity: 'critical',
  channels: ['slack', 'pagerduty'],
});

// Subscribe to alerts
alerting.on('alert-fired', (alert) => {
  console.log(`Alert: ${alert.rule.id} - ${alert.message}`);
});
```

## Troubleshooting

### Common Issues

**Job stuck in "provisioning" state**

- Check provider API credentials
- Verify GPU availability in the selected region
- Check for rate limiting

**Cost estimate significantly different from actual**

- Large file transfers can add unexpected costs
- Cold start times vary by provider
- Check for multi-GPU jobs

**Job failed with "GPU out of memory"**

- Increase `minVRAM` requirement
- Use a GPU with more memory (A100-80GB, H100)
- Reduce batch size in job parameters

**Provider unavailable**

- Check provider status page
- Verify API key is valid
- Check network connectivity

### Debug Logging

```typescript
import { setLogLevel } from '@yemaya/gpu-jobs';

// Enable debug logging
setLogLevel('debug');

// Or for specific components
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');
```

### Health Checks

```typescript
// Check system health
const health = await dispatcher.healthCheck();
console.log(`System Status: ${health.status}`);
console.log(`Providers:`, health.providers);
console.log(`Queue Depth: ${health.queueDepth}`);
```

## Advanced Features

### Health Dashboard

The health dashboard provides a comprehensive view of system status:

```typescript
// Get full health dashboard
const health = await gpuJobsService.getHealthDashboard({
  includeAlerts: true,
  includeProviderDetails: true,
  includeComparison: true,
  includeSystemMetrics: true,
});

console.log('Status:', health.status); // 'healthy' | 'degraded' | 'unhealthy' | 'critical'
console.log('Active Alerts:', health.alerts);
console.log('Provider Comparison:', health.comparison);
console.log('Queue Metrics:', health.queue);
console.log('Budget Status:', health.budget);
```

**API Endpoint:**

```
GET /api/v1/gpu-jobs/health
```

### A/B Testing

Compare provider performance with A/B tests:

```typescript
// Create A/B test
const test = await gpuJobsService.createABTest({
  name: 'RunPod vs Modal Performance',
  description: 'Compare latency and cost for texture upscaling',
  controlProvider: 'runpod-serverless',
  treatmentProvider: 'modal',
  trafficSplit: 50,
  targetSampleSize: 100,
  jobTypes: ['texture-upscale'],
});

// Get test results with statistical significance
const results = await gpuJobsService.getABTest(test.id);
console.log('Statistical Significance:', results.statisticalSignificance);
console.log('Recommended Winner:', results.winner);
```

**API Endpoints:**

```
POST   /api/v1/gpu-jobs/ab-tests           # Create A/B test
GET    /api/v1/gpu-jobs/ab-tests           # List A/B tests
GET    /api/v1/gpu-jobs/ab-tests/{testId}  # Get A/B test details
POST   /api/v1/gpu-jobs/ab-tests/{testId}/stop    # Stop A/B test
DELETE /api/v1/gpu-jobs/ab-tests/{testId}  # Delete A/B test
```

### Circuit Breakers

Automatic provider isolation with circuit breakers:

```typescript
// Get circuit breaker status
const status =
  await gpuJobsService.getCircuitBreakerStatus('runpod-serverless');
console.log('State:', status.state); // 'closed' | 'open' | 'half-open'
console.log('Failure Count:', status.failureCount);
console.log('Success Rate:', status.successRate);

// Reset circuit breaker
await gpuJobsService.resetCircuitBreaker('runpod-serverless');

// Update configuration
await gpuJobsService.updateCircuitBreakerConfig('runpod-serverless', {
  failureThreshold: 5,
  resetTimeoutMs: 60000,
  halfOpenMaxRequests: 3,
});
```

**API Endpoints:**

```
GET  /api/v1/gpu-jobs/circuit-breakers                    # List all circuit breakers
GET  /api/v1/gpu-jobs/circuit-breakers/{provider}         # Get provider status
POST /api/v1/gpu-jobs/circuit-breakers/{provider}/reset   # Reset circuit breaker
PATCH /api/v1/gpu-jobs/circuit-breakers/{provider}/config # Update config
```

### Dead Letter Queue (DLQ)

Manage permanently failed jobs:

```typescript
// List DLQ entries
const dlq = await gpuJobsService.listDLQ({
  failureReason: 'max_retries_exceeded',
  jobType: 'texture-upscale',
  limit: 20,
  sort: 'failedAt',
  order: 'desc',
});

// Get DLQ statistics
const stats = await gpuJobsService.getDLQStats();
console.log('Total Entries:', stats.totalEntries);
console.log('By Failure Reason:', stats.byFailureReason);
console.log('Total Cost Lost:', stats.totalCostLost);

// Replay a failed job
const result = await gpuJobsService.replayDLQEntry(entryId, {
  preferredProvider: 'modal', // Try different provider
  modifyParameters: { quality: 'high' },
});

// Bulk replay
const bulkResult = await gpuJobsService.replayDLQEntries(entryIds, {
  maxCost: 1.0,
});

// Purge old entries (with dry run)
const purgeResult = await gpuJobsService.purgeDLQ({
  olderThan: '2024-01-01',
  includeReplayed: true,
  dryRun: true, // Preview what would be purged
});
```

**API Endpoints:**

```
GET    /api/v1/gpu-jobs/dlq                    # List DLQ entries
GET    /api/v1/gpu-jobs/dlq/stats              # Get DLQ statistics
GET    /api/v1/gpu-jobs/dlq/{entryId}          # Get DLQ entry details
POST   /api/v1/gpu-jobs/dlq/{entryId}/replay   # Replay single entry
POST   /api/v1/gpu-jobs/dlq/replay             # Bulk replay entries
DELETE /api/v1/gpu-jobs/dlq/{entryId}          # Delete DLQ entry
POST   /api/v1/gpu-jobs/dlq/purge              # Purge DLQ entries
```

### Routing Rules Configuration

Configure job routing without code changes using JSON configuration:

```typescript
import { RoutingRulesLoader, RoutingRulesEngine } from '@yemaya/gpu-jobs';

// Load from file
const config = RoutingRulesLoader.loadFromFile('./routing-config.json');
const engine = new RoutingRulesEngine(config);

// Evaluate rules for a job
const decision = engine.evaluateRules(jobRequest, {
  currentCapacity: providerCapacities,
  estimatedCosts: costEstimates,
});

console.log('Provider Order:', decision.providerOrder);
console.log('Matched Rules:', decision.matchedRules);
console.log('Applied Settings:', decision.appliedSettings);
```

See [Routing Rules Guide](./routing-rules.md) for detailed configuration
options.

### Prometheus Metrics

Export metrics in Prometheus format:

```
GET /api/v1/gpu-jobs/metrics
```

**Available Metrics:**

- `gpu_jobs_total` - Total jobs by type, status, and provider
- `gpu_jobs_cost_total` - Total cost incurred
- `gpu_jobs_duration_seconds` - Job duration histogram
- `gpu_jobs_queue_depth` - Current queue depth
- `gpu_jobs_provider_success_rate` - Provider success rates
- `gpu_jobs_circuit_breaker_state` - Circuit breaker states
- `gpu_jobs_dlq_entries` - Dead letter queue size

## Related Documentation

- [API Reference](./api.md)
- [Provider Configuration](./providers.md)
- [Cost Model](./cost-model.md)
- [Routing Rules](./routing-rules.md)
- [Deployment Guide](./deployment.md)
- [Troubleshooting Guide](./troubleshooting.md)
