Eve library

GPU Provider Configuration

Guide

Revision 1

On this page

This guide covers configuration for all supported GPU cloud providers.

Table of Contents#

Provider Comparison#

Provider Best For Cold Start Min Billing Spot Support GPU Types
RunPod Serverless Short jobs, variable load ~3s 1s No RTX 3090, RTX 4090, A10, A100
RunPod Pod Long jobs, consistent load ~30s 60s No RTX 3090, RTX 4090, A10, A100, H100
AWS ECS Spot Cost-sensitive, interruptible ~120s 60s Yes T4, A10, A100
AWS ECS On-Demand Critical workloads ~120s 60s No T4, A10, A100
Modal Python workloads, rapid dev ~10s 1s No T4, L4, A10, A100, H100
Vast.ai Budget-conscious, variable ~60s 1s Yes Consumer & Data Center
Local Development, testing 0s N/A N/A Any installed

RunPod Serverless#

Overview#

RunPod Serverless provides pay-per-second GPU compute with minimal cold start times. Ideal for short-duration, variable workloads.

Setup#

  1. Create a RunPod account at runpod.io
  2. Generate an API key in Account Settings
  3. Create a Serverless Endpoint with your worker image

Configuration#

typescript
import { createRunPodServerlessExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createRunPodServerlessExecutor({
  // Required
  apiKey: process.env.RUNPOD_API_KEY!,
  endpointId: process.env.RUNPOD_ENDPOINT_ID!,

  // Optional
  baseUrl: 'https://api.runpod.ai/v2', // Default
  timeout: 300000, // Request timeout (ms)
  maxRetries: 3, // Retry count

  // Worker configuration
  workerConfig: {
    minWorkers: 0, // Min warm workers
    maxWorkers: 10, // Max concurrent workers
    idleTimeout: 60, // Idle timeout (seconds)
    flashBoot: true, // Enable flash boot
  },
});

Environment Variables#

bash
RUNPOD_API_KEY=your_api_key_here
RUNPOD_ENDPOINT_ID=your_endpoint_id

Worker Image Requirements#

Your RunPod worker image should:

  • Accept jobs via the RunPod handler format
  • Return results in the expected format
  • Handle graceful shutdown signals
python
# Example RunPod handler
import runpod

def handler(event):
    job_input = event["input"]

    # Process the job
    result = process_gpu_job(job_input)

    return {"output": result}

runpod.serverless.start({"handler": handler})

Pricing (as of 2024)#

GPU Price/Second Price/Hour
RTX 3090 $0.00044 $1.58
RTX 4090 $0.00069 $2.48
A10 $0.00058 $2.09
A100 40GB $0.00139 $5.00
A100 80GB $0.00222 $7.99

RunPod Pod#

Overview#

RunPod Pods are persistent GPU instances that remain running until explicitly stopped. Better economics for long-running workloads.

Setup#

  1. Create a Pod Template in RunPod dashboard
  2. Configure the template with your worker image
  3. Note the template ID

Configuration#

typescript
import { createRunPodPodExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createRunPodPodExecutor({
  // Required
  apiKey: process.env.RUNPOD_API_KEY!,
  templateId: process.env.RUNPOD_TEMPLATE_ID!,

  // Optional
  baseUrl: 'https://api.runpod.ai/v2',

  // Pod configuration
  podConfig: {
    cloudType: 'COMMUNITY', // or 'SECURE'
    gpuCount: 1,
    volumeInGb: 50, // Persistent volume size
    containerDiskInGb: 20, // Container disk size
    minVcpuCount: 4,
    minMemoryInGb: 16,
    ports: '8080/http,22/tcp', // Exposed ports
    startSsh: true,
  },

  // Lifecycle
  autoStop: true, // Stop pod when idle
  idleTimeout: 300, // Idle timeout (seconds)
});

Environment Variables#

bash
RUNPOD_API_KEY=your_api_key_here
RUNPOD_TEMPLATE_ID=your_template_id

Pricing (as of 2024)#

GPU Price/Hour (Community) Price/Hour (Secure)
RTX 3090 $0.69 $0.89
RTX 4090 $1.14 $1.44
A10 $1.00 $1.30
A100 40GB $3.19 $3.99
A100 80GB $5.19 $6.49
H100 $14.00 $17.00

AWS ECS#

Overview#

AWS ECS (Elastic Container Service) with GPU support for enterprise workloads integrated with AWS ecosystem.

Prerequisites#

  1. AWS account with ECS permissions
  2. GPU-enabled EC2 launch template
  3. ECS cluster with GPU capacity providers
  4. Task definition with GPU resource requirements

Configuration#

typescript
import { createAWSECSExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createAWSECSExecutor({
  // Required
  region: 'us-east-1',
  cluster: 'gpu-jobs-cluster',
  taskDefinition: 'gpu-worker:latest',

  // Optional
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },

  // Spot configuration
  useSpot: true, // Use Spot instances
  spotMaxPrice: '0.50', // Max spot price per hour
  spotAllocationStrategy: 'lowest-price',

  // Network configuration
  subnets: ['subnet-xxx', 'subnet-yyy'],
  securityGroups: ['sg-xxx'],
  assignPublicIp: true,

  // Capacity provider
  capacityProvider: 'gpu-spot-provider',

  // Task configuration
  taskOverrides: {
    cpu: '4096',
    memory: '16384',
    resourceRequirements: [{ type: 'GPU', value: '1' }],
  },
});

Environment Variables#

bash
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=us-east-1

ECS Task Definition Example#

json
{
  "family": "gpu-worker",
  "requiresCompatibilities": ["EC2"],
  "cpu": "4096",
  "memory": "16384",
  "containerDefinitions": [
    {
      "name": "gpu-worker",
      "image": "your-registry/gpu-worker:latest",
      "resourceRequirements": [
        {
          "type": "GPU",
          "value": "1"
        }
      ],
      "environment": [{ "name": "NVIDIA_VISIBLE_DEVICES", "value": "all" }],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/gpu-worker",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

Pricing (as of 2024)#

Instance GPU On-Demand/Hour Spot/Hour (avg)
g4dn.xlarge T4 $0.526 $0.16
g5.xlarge A10 $1.01 $0.30
p4d.24xlarge A100 x8 $32.77 $9.83

Overview#

Modal provides a Python-first serverless GPU platform with excellent developer experience and per-second billing.

Setup#

  1. Install Modal: pip install modal
  2. Authenticate: modal token new
  3. Create an app in the Modal dashboard

Configuration#

typescript
import { createModalExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createModalExecutor({
  // Required
  token: process.env.MODAL_TOKEN!,
  appName: 'gpu-jobs',

  // Optional
  workspace: 'default',

  // Function configuration
  functionConfig: {
    gpu: 'A100', // GPU type
    memory: 16384, // Memory in MB
    timeout: 3600, // Timeout in seconds
    retries: 3,
    concurrencyLimit: 10,
  },

  // Image configuration
  imageConfig: {
    pythonVersion: '3.10',
    pipPackages: ['torch', 'transformers'],
    dockerfileCommands: [],
  },
});

Environment Variables#

bash
MODAL_TOKEN=your_modal_token
python
import modal

app = modal.App("gpu-jobs")

@app.function(
    gpu="A100",
    memory=16384,
    timeout=3600,
)
def process_gpu_job(job_input: dict) -> dict:
    import torch

    # Your GPU processing logic
    result = run_inference(job_input)

    return {"output": result}

Pricing (as of 2024)#

GPU Price/Second Price/Hour
T4 $0.000164 $0.59
L4 $0.000306 $1.10
A10 $0.000306 $1.10
A100 40GB $0.001078 $3.88
A100 80GB $0.001564 $5.63
H100 $0.002256 $8.12

Vast.ai#

Overview#

Vast.ai is a marketplace for GPU compute, offering some of the lowest prices but with variable availability.

Configuration#

typescript
import { createVastAIExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createVastAIExecutor({
  // Required
  apiKey: process.env.VASTAI_API_KEY!,

  // Search criteria
  searchCriteria: {
    gpuName: ['RTX 4090', 'A100'],
    minGpuRam: 24,
    minCpuRam: 32,
    minDiskSpace: 50,
    reliability: 0.95, // Min reliability score
    dlPerf: 10, // Min download performance
    dphTotal: 2.0, // Max $/hour total
  },

  // Instance configuration
  instanceConfig: {
    image: 'your-docker-image:latest',
    diskSpace: 50,
    onstart: '/start.sh',
    envVars: { CUDA_VISIBLE_DEVICES: '0' },
  },
});

Environment Variables#

bash
VASTAI_API_KEY=your_vastai_api_key

Pricing#

Vast.ai pricing is dynamic and depends on the marketplace. Typical ranges:

GPU Price/Hour Range
RTX 3090 $0.30 - $0.60
RTX 4090 $0.50 - $0.90
A100 40GB $1.00 - $2.00
A100 80GB $1.50 - $3.00

Local GPU#

Overview#

Use local GPUs for development, testing, or on-premise deployments.

Configuration#

typescript
import { createLocalExecutor } from '@yemaya/gpu-jobs/providers';

const executor = createLocalExecutor({
  // GPU configuration
  gpuIds: [0, 1], // Available GPU indices
  maxConcurrent: 2, // Max concurrent jobs

  // Worker configuration
  workerScript: './worker.py', // Path to worker script
  pythonPath: '/usr/bin/python3', // Python executable

  // Resource limits
  maxMemoryPerJob: 16 * 1024, // Max memory per job (MB)
  maxDiskPerJob: 50 * 1024, // Max disk per job (MB)

  // Paths
  workDir: '/tmp/gpu-jobs', // Working directory
  outputDir: '/data/outputs', // Output directory
});

GPU Detection#

typescript
// Auto-detect available GPUs
const gpus = await localExecutor.detectGPUs();
console.log('Available GPUs:', gpus);
// [
//   { id: 0, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 0 },
//   { id: 1, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 15 },
// ]

Multi-Provider Setup#

Configuration#

typescript
import {
  createJobDispatcher,
  createGPUCostEstimator,
  createRunPodServerlessExecutor,
  createRunPodPodExecutor,
  createModalExecutor,
  createLocalExecutor,
} from '@yemaya/gpu-jobs';

const dispatcher = createJobDispatcher({
  costEstimator: createGPUCostEstimator(),

  providers: {
    'runpod-serverless': createRunPodServerlessExecutor({
      apiKey: process.env.RUNPOD_API_KEY!,
      endpointId: process.env.RUNPOD_ENDPOINT_ID!,
    }),
    'runpod-pod': createRunPodPodExecutor({
      apiKey: process.env.RUNPOD_API_KEY!,
      templateId: process.env.RUNPOD_TEMPLATE_ID!,
    }),
    modal: createModalExecutor({
      token: process.env.MODAL_TOKEN!,
      appName: 'gpu-jobs',
    }),
    local: createLocalExecutor({
      gpuIds: [0],
      maxConcurrent: 1,
    }),
  },

  // Provider selection strategy
  selectionStrategy: 'cheapest', // 'cheapest', 'fastest', 'most-available'

  // Default provider
  defaultProvider: 'runpod-serverless',
});

Selection Strategies#

Strategy Description
cheapest Select provider with lowest estimated cost
fastest Select provider with shortest estimated duration (including cold start)
most-available Select provider with most available capacity
round-robin Distribute jobs evenly across providers
priority Use providers in configured priority order

Failover Configuration#

Automatic Failover#

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  providers: {
    /* ... */
  },

  failoverConfig: {
    enabled: true,
    maxAttempts: 3, // Max failover attempts
    providers: [
      // Failover order
      'runpod-serverless',
      'modal',
      'runpod-pod',
    ],

    // Conditions for failover
    failoverOn: [
      'provider-unavailable',
      'capacity-exhausted',
      'timeout',
      'rate-limited',
    ],

    // Don't failover for these errors
    noFailoverOn: ['invalid-input', 'budget-exceeded', 'authentication-failed'],
  },
});

Manual Failover#

typescript
// Force job to specific provider
const result = await dispatcher.dispatch(job, {
  forceProvider: 'modal',
});

// Retry on different provider
try {
  await dispatcher.dispatch(job);
} catch (error) {
  if (error.code === 'PROVIDER_UNAVAILABLE') {
    await dispatcher.dispatch(job, {
      excludeProviders: [error.provider],
    });
  }
}

Health-Based Routing#

typescript
const dispatcher = createJobDispatcher({
  costEstimator,
  providers: {
    /* ... */
  },

  healthConfig: {
    checkInterval: 30000, // Health check interval (ms)
    unhealthyThreshold: 3, // Failures before unhealthy
    healthyThreshold: 2, // Successes before healthy

    // Route away from unhealthy providers
    routeAroundUnhealthy: true,
  },
});