# GPU Provider Configuration

This guide covers configuration for all supported GPU cloud providers.

## Table of Contents

- [Provider Comparison](#provider-comparison)
- [RunPod Serverless](#runpod-serverless)
- [RunPod Pod](#runpod-pod)
- [AWS ECS](#aws-ecs)
- [Modal](#modal)
- [Vast.ai](#vastai)
- [Local GPU](#local-gpu)
- [Multi-Provider Setup](#multi-provider-setup)
- [Failover Configuration](#failover-configuration)

## Provider Comparison

| Provider          | Best For                      | Cold Start | Min Billing | Spot Support | GPU Types                           |
| ----------------- | ----------------------------- | ---------- | ----------- | ------------ | ----------------------------------- |
| RunPod Serverless | Short jobs, variable load     | ~3s        | 1s          | No           | RTX 3090, RTX 4090, A10, A100       |
| RunPod Pod        | Long jobs, consistent load    | ~30s       | 60s         | No           | RTX 3090, RTX 4090, A10, A100, H100 |
| AWS ECS Spot      | Cost-sensitive, interruptible | ~120s      | 60s         | Yes          | T4, A10, A100                       |
| AWS ECS On-Demand | Critical workloads            | ~120s      | 60s         | No           | T4, A10, A100                       |
| Modal             | Python workloads, rapid dev   | ~10s       | 1s          | No           | T4, L4, A10, A100, H100             |
| Vast.ai           | Budget-conscious, variable    | ~60s       | 1s          | Yes          | Consumer & Data Center              |
| Local             | Development, testing          | 0s         | N/A         | N/A          | Any installed                       |

## RunPod Serverless

### Overview

RunPod Serverless provides pay-per-second GPU compute with minimal cold start
times. Ideal for short-duration, variable workloads.

### Setup

1. Create a RunPod account at [runpod.io](https://runpod.io)
2. Generate an API key in Account Settings
3. Create a Serverless Endpoint with your worker image

### Configuration

```typescript
import { createRunPodServerlessExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createRunPodServerlessExecutor({
  // Required
  apiKey: process.env.RUNPOD_API_KEY!,
  endpointId: process.env.RUNPOD_ENDPOINT_ID!,

  // Optional
  baseUrl: 'https://api.runpod.ai/v2', // Default
  timeout: 300000, // Request timeout (ms)
  maxRetries: 3, // Retry count

  // Worker configuration
  workerConfig: {
    minWorkers: 0, // Min warm workers
    maxWorkers: 10, // Max concurrent workers
    idleTimeout: 60, // Idle timeout (seconds)
    flashBoot: true, // Enable flash boot
  },
});
```

### Environment Variables

```bash
RUNPOD_API_KEY=your_api_key_here
RUNPOD_ENDPOINT_ID=your_endpoint_id
```

### Worker Image Requirements

Your RunPod worker image should:

- Accept jobs via the RunPod handler format
- Return results in the expected format
- Handle graceful shutdown signals

```python
# Example RunPod handler
import runpod

def handler(event):
    job_input = event["input"]

    # Process the job
    result = process_gpu_job(job_input)

    return {"output": result}

runpod.serverless.start({"handler": handler})
```

### Pricing (as of 2024)

| GPU       | Price/Second | Price/Hour |
| --------- | ------------ | ---------- |
| RTX 3090  | $0.00044     | $1.58      |
| RTX 4090  | $0.00069     | $2.48      |
| A10       | $0.00058     | $2.09      |
| A100 40GB | $0.00139     | $5.00      |
| A100 80GB | $0.00222     | $7.99      |

## RunPod Pod

### Overview

RunPod Pods are persistent GPU instances that remain running until explicitly
stopped. Better economics for long-running workloads.

### Setup

1. Create a Pod Template in RunPod dashboard
2. Configure the template with your worker image
3. Note the template ID

### Configuration

```typescript
import { createRunPodPodExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createRunPodPodExecutor({
  // Required
  apiKey: process.env.RUNPOD_API_KEY!,
  templateId: process.env.RUNPOD_TEMPLATE_ID!,

  // Optional
  baseUrl: 'https://api.runpod.ai/v2',

  // Pod configuration
  podConfig: {
    cloudType: 'COMMUNITY', // or 'SECURE'
    gpuCount: 1,
    volumeInGb: 50, // Persistent volume size
    containerDiskInGb: 20, // Container disk size
    minVcpuCount: 4,
    minMemoryInGb: 16,
    ports: '8080/http,22/tcp', // Exposed ports
    startSsh: true,
  },

  // Lifecycle
  autoStop: true, // Stop pod when idle
  idleTimeout: 300, // Idle timeout (seconds)
});
```

### Environment Variables

```bash
RUNPOD_API_KEY=your_api_key_here
RUNPOD_TEMPLATE_ID=your_template_id
```

### Pricing (as of 2024)

| GPU       | Price/Hour (Community) | Price/Hour (Secure) |
| --------- | ---------------------- | ------------------- |
| RTX 3090  | $0.69                  | $0.89               |
| RTX 4090  | $1.14                  | $1.44               |
| A10       | $1.00                  | $1.30               |
| A100 40GB | $3.19                  | $3.99               |
| A100 80GB | $5.19                  | $6.49               |
| H100      | $14.00                 | $17.00              |

## AWS ECS

### Overview

AWS ECS (Elastic Container Service) with GPU support for enterprise workloads
integrated with AWS ecosystem.

### Prerequisites

1. AWS account with ECS permissions
2. GPU-enabled EC2 launch template
3. ECS cluster with GPU capacity providers
4. Task definition with GPU resource requirements

### Configuration

```typescript
import { createAWSECSExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createAWSECSExecutor({
  // Required
  region: 'us-east-1',
  cluster: 'gpu-jobs-cluster',
  taskDefinition: 'gpu-worker:latest',

  // Optional
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },

  // Spot configuration
  useSpot: true, // Use Spot instances
  spotMaxPrice: '0.50', // Max spot price per hour
  spotAllocationStrategy: 'lowest-price',

  // Network configuration
  subnets: ['subnet-xxx', 'subnet-yyy'],
  securityGroups: ['sg-xxx'],
  assignPublicIp: true,

  // Capacity provider
  capacityProvider: 'gpu-spot-provider',

  // Task configuration
  taskOverrides: {
    cpu: '4096',
    memory: '16384',
    resourceRequirements: [{ type: 'GPU', value: '1' }],
  },
});
```

### Environment Variables

```bash
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=us-east-1
```

### ECS Task Definition Example

```json
{
  "family": "gpu-worker",
  "requiresCompatibilities": ["EC2"],
  "cpu": "4096",
  "memory": "16384",
  "containerDefinitions": [
    {
      "name": "gpu-worker",
      "image": "your-registry/gpu-worker:latest",
      "resourceRequirements": [
        {
          "type": "GPU",
          "value": "1"
        }
      ],
      "environment": [{ "name": "NVIDIA_VISIBLE_DEVICES", "value": "all" }],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/gpu-worker",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}
```

### Pricing (as of 2024)

| Instance     | GPU     | On-Demand/Hour | Spot/Hour (avg) |
| ------------ | ------- | -------------- | --------------- |
| g4dn.xlarge  | T4      | $0.526         | $0.16           |
| g5.xlarge    | A10     | $1.01          | $0.30           |
| p4d.24xlarge | A100 x8 | $32.77         | $9.83           |

## Modal

### Overview

Modal provides a Python-first serverless GPU platform with excellent developer
experience and per-second billing.

### Setup

1. Install Modal: `pip install modal`
2. Authenticate: `modal token new`
3. Create an app in the Modal dashboard

### Configuration

```typescript
import { createModalExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createModalExecutor({
  // Required
  token: process.env.MODAL_TOKEN!,
  appName: 'gpu-jobs',

  // Optional
  workspace: 'default',

  // Function configuration
  functionConfig: {
    gpu: 'A100', // GPU type
    memory: 16384, // Memory in MB
    timeout: 3600, // Timeout in seconds
    retries: 3,
    concurrencyLimit: 10,
  },

  // Image configuration
  imageConfig: {
    pythonVersion: '3.10',
    pipPackages: ['torch', 'transformers'],
    dockerfileCommands: [],
  },
});
```

### Environment Variables

```bash
MODAL_TOKEN=your_modal_token
```

### Modal Function Example

```python
import modal

app = modal.App("gpu-jobs")

@app.function(
    gpu="A100",
    memory=16384,
    timeout=3600,
)
def process_gpu_job(job_input: dict) -> dict:
    import torch

    # Your GPU processing logic
    result = run_inference(job_input)

    return {"output": result}
```

### Pricing (as of 2024)

| GPU       | Price/Second | Price/Hour |
| --------- | ------------ | ---------- |
| T4        | $0.000164    | $0.59      |
| L4        | $0.000306    | $1.10      |
| A10       | $0.000306    | $1.10      |
| A100 40GB | $0.001078    | $3.88      |
| A100 80GB | $0.001564    | $5.63      |
| H100      | $0.002256    | $8.12      |

## Vast.ai

### Overview

Vast.ai is a marketplace for GPU compute, offering some of the lowest prices but
with variable availability.

### Configuration

```typescript
import { createVastAIExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createVastAIExecutor({
  // Required
  apiKey: process.env.VASTAI_API_KEY!,

  // Search criteria
  searchCriteria: {
    gpuName: ['RTX 4090', 'A100'],
    minGpuRam: 24,
    minCpuRam: 32,
    minDiskSpace: 50,
    reliability: 0.95, // Min reliability score
    dlPerf: 10, // Min download performance
    dphTotal: 2.0, // Max $/hour total
  },

  // Instance configuration
  instanceConfig: {
    image: 'your-docker-image:latest',
    diskSpace: 50,
    onstart: '/start.sh',
    envVars: { CUDA_VISIBLE_DEVICES: '0' },
  },
});
```

### Environment Variables

```bash
VASTAI_API_KEY=your_vastai_api_key
```

### Pricing

Vast.ai pricing is dynamic and depends on the marketplace. Typical ranges:

| GPU       | Price/Hour Range |
| --------- | ---------------- |
| RTX 3090  | $0.30 - $0.60    |
| RTX 4090  | $0.50 - $0.90    |
| A100 40GB | $1.00 - $2.00    |
| A100 80GB | $1.50 - $3.00    |

## Local GPU

### Overview

Use local GPUs for development, testing, or on-premise deployments.

### Configuration

```typescript
import { createLocalExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createLocalExecutor({
  // GPU configuration
  gpuIds: [0, 1], // Available GPU indices
  maxConcurrent: 2, // Max concurrent jobs

  // Worker configuration
  workerScript: './worker.py', // Path to worker script
  pythonPath: '/usr/bin/python3', // Python executable

  // Resource limits
  maxMemoryPerJob: 16 * 1024, // Max memory per job (MB)
  maxDiskPerJob: 50 * 1024, // Max disk per job (MB)

  // Paths
  workDir: '/tmp/gpu-jobs', // Working directory
  outputDir: '/data/outputs', // Output directory
});
```

### GPU Detection

```typescript
// Auto-detect available GPUs
const gpus = await localExecutor.detectGPUs();
console.log('Available GPUs:', gpus);
// [
//   { id: 0, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 0 },
//   { id: 1, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 15 },
// ]
```

## Multi-Provider Setup

### Configuration

```typescript
import {
  createJobDispatcher,
  createGPUCostEstimator,
  createRunPodServerlessExecutor,
  createRunPodPodExecutor,
  createModalExecutor,
  createLocalExecutor,
} from '@yemaya/gpu-jobs';

const dispatcher = createJobDispatcher({
  costEstimator: createGPUCostEstimator(),

  providers: {
    'runpod-serverless': createRunPodServerlessExecutor({
      apiKey: process.env.RUNPOD_API_KEY!,
      endpointId: process.env.RUNPOD_ENDPOINT_ID!,
    }),
    'runpod-pod': createRunPodPodExecutor({
      apiKey: process.env.RUNPOD_API_KEY!,
      templateId: process.env.RUNPOD_TEMPLATE_ID!,
    }),
    modal: createModalExecutor({
      token: process.env.MODAL_TOKEN!,
      appName: 'gpu-jobs',
    }),
    local: createLocalExecutor({
      gpuIds: [0],
      maxConcurrent: 1,
    }),
  },

  // Provider selection strategy
  selectionStrategy: 'cheapest', // 'cheapest', 'fastest', 'most-available'

  // Default provider
  defaultProvider: 'runpod-serverless',
});
```

### Selection Strategies

| Strategy         | Description                                                             |
| ---------------- | ----------------------------------------------------------------------- |
| `cheapest`       | Select provider with lowest estimated cost                              |
| `fastest`        | Select provider with shortest estimated duration (including cold start) |
| `most-available` | Select provider with most available capacity                            |
| `round-robin`    | Distribute jobs evenly across providers                                 |
| `priority`       | Use providers in configured priority order                              |

## Failover Configuration

### Automatic Failover

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  providers: {
    /* ... */
  },

  failoverConfig: {
    enabled: true,
    maxAttempts: 3, // Max failover attempts
    providers: [
      // Failover order
      'runpod-serverless',
      'modal',
      'runpod-pod',
    ],

    // Conditions for failover
    failoverOn: [
      'provider-unavailable',
      'capacity-exhausted',
      'timeout',
      'rate-limited',
    ],

    // Don't failover for these errors
    noFailoverOn: ['invalid-input', 'budget-exceeded', 'authentication-failed'],
  },
});
```

### Manual Failover

```typescript
// Force job to specific provider
const result = await dispatcher.dispatch(job, {
  forceProvider: 'modal',
});

// Retry on different provider
try {
  await dispatcher.dispatch(job);
} catch (error) {
  if (error.code === 'PROVIDER_UNAVAILABLE') {
    await dispatcher.dispatch(job, {
      excludeProviders: [error.provider],
    });
  }
}
```

### Health-Based Routing

```typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  providers: {
    /* ... */
  },

  healthConfig: {
    checkInterval: 30000, // Health check interval (ms)
    unhealthyThreshold: 3, // Failures before unhealthy
    healthyThreshold: 2, // Successes before healthy

    // Route away from unhealthy providers
    routeAroundUnhealthy: true,
  },
});
```
