This guide covers configuration for all supported GPU cloud providers.
Table of Contents#
- Provider Comparison
- RunPod Serverless
- RunPod Pod
- AWS ECS
- Modal
- Vast.ai
- Local GPU
- Multi-Provider Setup
- Failover Configuration
Provider Comparison#
| Provider | Best For | Cold Start | Min Billing | Spot Support | GPU Types |
|---|---|---|---|---|---|
| RunPod Serverless | Short jobs, variable load | ~3s | 1s | No | RTX 3090, RTX 4090, A10, A100 |
| RunPod Pod | Long jobs, consistent load | ~30s | 60s | No | RTX 3090, RTX 4090, A10, A100, H100 |
| AWS ECS Spot | Cost-sensitive, interruptible | ~120s | 60s | Yes | T4, A10, A100 |
| AWS ECS On-Demand | Critical workloads | ~120s | 60s | No | T4, A10, A100 |
| Modal | Python workloads, rapid dev | ~10s | 1s | No | T4, L4, A10, A100, H100 |
| Vast.ai | Budget-conscious, variable | ~60s | 1s | Yes | Consumer & Data Center |
| Local | Development, testing | 0s | N/A | N/A | Any installed |
RunPod Serverless#
Overview#
RunPod Serverless provides pay-per-second GPU compute with minimal cold start times. Ideal for short-duration, variable workloads.
Setup#
- Create a RunPod account at runpod.io
- Generate an API key in Account Settings
- Create a Serverless Endpoint with your worker image
Configuration#
import { createRunPodServerlessExecutor } from '@yemaya/gpu-jobs/providers';
const executor = createRunPodServerlessExecutor({
// Required
apiKey: process.env.RUNPOD_API_KEY!,
endpointId: process.env.RUNPOD_ENDPOINT_ID!,
// Optional
baseUrl: 'https://api.runpod.ai/v2', // Default
timeout: 300000, // Request timeout (ms)
maxRetries: 3, // Retry count
// Worker configuration
workerConfig: {
minWorkers: 0, // Min warm workers
maxWorkers: 10, // Max concurrent workers
idleTimeout: 60, // Idle timeout (seconds)
flashBoot: true, // Enable flash boot
},
});
Environment Variables#
RUNPOD_API_KEY=your_api_key_here
RUNPOD_ENDPOINT_ID=your_endpoint_id
Worker Image Requirements#
Your RunPod worker image should:
- Accept jobs via the RunPod handler format
- Return results in the expected format
- Handle graceful shutdown signals
# Example RunPod handler
import runpod
def handler(event):
job_input = event["input"]
# Process the job
result = process_gpu_job(job_input)
return {"output": result}
runpod.serverless.start({"handler": handler})
Pricing (as of 2024)#
| GPU | Price/Second | Price/Hour |
|---|---|---|
| RTX 3090 | $0.00044 | $1.58 |
| RTX 4090 | $0.00069 | $2.48 |
| A10 | $0.00058 | $2.09 |
| A100 40GB | $0.00139 | $5.00 |
| A100 80GB | $0.00222 | $7.99 |
RunPod Pod#
Overview#
RunPod Pods are persistent GPU instances that remain running until explicitly stopped. Better economics for long-running workloads.
Setup#
- Create a Pod Template in RunPod dashboard
- Configure the template with your worker image
- Note the template ID
Configuration#
import { createRunPodPodExecutor } from '@yemaya/gpu-jobs/providers';
const executor = createRunPodPodExecutor({
// Required
apiKey: process.env.RUNPOD_API_KEY!,
templateId: process.env.RUNPOD_TEMPLATE_ID!,
// Optional
baseUrl: 'https://api.runpod.ai/v2',
// Pod configuration
podConfig: {
cloudType: 'COMMUNITY', // or 'SECURE'
gpuCount: 1,
volumeInGb: 50, // Persistent volume size
containerDiskInGb: 20, // Container disk size
minVcpuCount: 4,
minMemoryInGb: 16,
ports: '8080/http,22/tcp', // Exposed ports
startSsh: true,
},
// Lifecycle
autoStop: true, // Stop pod when idle
idleTimeout: 300, // Idle timeout (seconds)
});
Environment Variables#
RUNPOD_API_KEY=your_api_key_here
RUNPOD_TEMPLATE_ID=your_template_id
Pricing (as of 2024)#
| GPU | Price/Hour (Community) | Price/Hour (Secure) |
|---|---|---|
| RTX 3090 | $0.69 | $0.89 |
| RTX 4090 | $1.14 | $1.44 |
| A10 | $1.00 | $1.30 |
| A100 40GB | $3.19 | $3.99 |
| A100 80GB | $5.19 | $6.49 |
| H100 | $14.00 | $17.00 |
AWS ECS#
Overview#
AWS ECS (Elastic Container Service) with GPU support for enterprise workloads integrated with AWS ecosystem.
Prerequisites#
- AWS account with ECS permissions
- GPU-enabled EC2 launch template
- ECS cluster with GPU capacity providers
- Task definition with GPU resource requirements
Configuration#
import { createAWSECSExecutor } from '@yemaya/gpu-jobs/providers';
const executor = createAWSECSExecutor({
// Required
region: 'us-east-1',
cluster: 'gpu-jobs-cluster',
taskDefinition: 'gpu-worker:latest',
// Optional
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
},
// Spot configuration
useSpot: true, // Use Spot instances
spotMaxPrice: '0.50', // Max spot price per hour
spotAllocationStrategy: 'lowest-price',
// Network configuration
subnets: ['subnet-xxx', 'subnet-yyy'],
securityGroups: ['sg-xxx'],
assignPublicIp: true,
// Capacity provider
capacityProvider: 'gpu-spot-provider',
// Task configuration
taskOverrides: {
cpu: '4096',
memory: '16384',
resourceRequirements: [{ type: 'GPU', value: '1' }],
},
});
Environment Variables#
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=us-east-1
ECS Task Definition Example#
{
"family": "gpu-worker",
"requiresCompatibilities": ["EC2"],
"cpu": "4096",
"memory": "16384",
"containerDefinitions": [
{
"name": "gpu-worker",
"image": "your-registry/gpu-worker:latest",
"resourceRequirements": [
{
"type": "GPU",
"value": "1"
}
],
"environment": [{ "name": "NVIDIA_VISIBLE_DEVICES", "value": "all" }],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/gpu-worker",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}
]
}
Pricing (as of 2024)#
| Instance | GPU | On-Demand/Hour | Spot/Hour (avg) |
|---|---|---|---|
| g4dn.xlarge | T4 | $0.526 | $0.16 |
| g5.xlarge | A10 | $1.01 | $0.30 |
| p4d.24xlarge | A100 x8 | $32.77 | $9.83 |
Modal#
Overview#
Modal provides a Python-first serverless GPU platform with excellent developer experience and per-second billing.
Setup#
- Install Modal:
pip install modal - Authenticate:
modal token new - Create an app in the Modal dashboard
Configuration#
import { createModalExecutor } from '@yemaya/gpu-jobs/providers';
const executor = createModalExecutor({
// Required
token: process.env.MODAL_TOKEN!,
appName: 'gpu-jobs',
// Optional
workspace: 'default',
// Function configuration
functionConfig: {
gpu: 'A100', // GPU type
memory: 16384, // Memory in MB
timeout: 3600, // Timeout in seconds
retries: 3,
concurrencyLimit: 10,
},
// Image configuration
imageConfig: {
pythonVersion: '3.10',
pipPackages: ['torch', 'transformers'],
dockerfileCommands: [],
},
});
Environment Variables#
MODAL_TOKEN=your_modal_token
Modal Function Example#
import modal
app = modal.App("gpu-jobs")
@app.function(
gpu="A100",
memory=16384,
timeout=3600,
)
def process_gpu_job(job_input: dict) -> dict:
import torch
# Your GPU processing logic
result = run_inference(job_input)
return {"output": result}
Pricing (as of 2024)#
| GPU | Price/Second | Price/Hour |
|---|---|---|
| T4 | $0.000164 | $0.59 |
| L4 | $0.000306 | $1.10 |
| A10 | $0.000306 | $1.10 |
| A100 40GB | $0.001078 | $3.88 |
| A100 80GB | $0.001564 | $5.63 |
| H100 | $0.002256 | $8.12 |
Vast.ai#
Overview#
Vast.ai is a marketplace for GPU compute, offering some of the lowest prices but with variable availability.
Configuration#
import { createVastAIExecutor } from '@yemaya/gpu-jobs/providers';
const executor = createVastAIExecutor({
// Required
apiKey: process.env.VASTAI_API_KEY!,
// Search criteria
searchCriteria: {
gpuName: ['RTX 4090', 'A100'],
minGpuRam: 24,
minCpuRam: 32,
minDiskSpace: 50,
reliability: 0.95, // Min reliability score
dlPerf: 10, // Min download performance
dphTotal: 2.0, // Max $/hour total
},
// Instance configuration
instanceConfig: {
image: 'your-docker-image:latest',
diskSpace: 50,
onstart: '/start.sh',
envVars: { CUDA_VISIBLE_DEVICES: '0' },
},
});
Environment Variables#
VASTAI_API_KEY=your_vastai_api_key
Pricing#
Vast.ai pricing is dynamic and depends on the marketplace. Typical ranges:
| GPU | Price/Hour Range |
|---|---|
| RTX 3090 | $0.30 - $0.60 |
| RTX 4090 | $0.50 - $0.90 |
| A100 40GB | $1.00 - $2.00 |
| A100 80GB | $1.50 - $3.00 |
Local GPU#
Overview#
Use local GPUs for development, testing, or on-premise deployments.
Configuration#
import { createLocalExecutor } from '@yemaya/gpu-jobs/providers';
const executor = createLocalExecutor({
// GPU configuration
gpuIds: [0, 1], // Available GPU indices
maxConcurrent: 2, // Max concurrent jobs
// Worker configuration
workerScript: './worker.py', // Path to worker script
pythonPath: '/usr/bin/python3', // Python executable
// Resource limits
maxMemoryPerJob: 16 * 1024, // Max memory per job (MB)
maxDiskPerJob: 50 * 1024, // Max disk per job (MB)
// Paths
workDir: '/tmp/gpu-jobs', // Working directory
outputDir: '/data/outputs', // Output directory
});
GPU Detection#
// Auto-detect available GPUs
const gpus = await localExecutor.detectGPUs();
console.log('Available GPUs:', gpus);
// [
// { id: 0, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 0 },
// { id: 1, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 15 },
// ]
Multi-Provider Setup#
Configuration#
import {
createJobDispatcher,
createGPUCostEstimator,
createRunPodServerlessExecutor,
createRunPodPodExecutor,
createModalExecutor,
createLocalExecutor,
} from '@yemaya/gpu-jobs';
const dispatcher = createJobDispatcher({
costEstimator: createGPUCostEstimator(),
providers: {
'runpod-serverless': createRunPodServerlessExecutor({
apiKey: process.env.RUNPOD_API_KEY!,
endpointId: process.env.RUNPOD_ENDPOINT_ID!,
}),
'runpod-pod': createRunPodPodExecutor({
apiKey: process.env.RUNPOD_API_KEY!,
templateId: process.env.RUNPOD_TEMPLATE_ID!,
}),
modal: createModalExecutor({
token: process.env.MODAL_TOKEN!,
appName: 'gpu-jobs',
}),
local: createLocalExecutor({
gpuIds: [0],
maxConcurrent: 1,
}),
},
// Provider selection strategy
selectionStrategy: 'cheapest', // 'cheapest', 'fastest', 'most-available'
// Default provider
defaultProvider: 'runpod-serverless',
});
Selection Strategies#
| Strategy | Description |
|---|---|
cheapest |
Select provider with lowest estimated cost |
fastest |
Select provider with shortest estimated duration (including cold start) |
most-available |
Select provider with most available capacity |
round-robin |
Distribute jobs evenly across providers |
priority |
Use providers in configured priority order |
Failover Configuration#
Automatic Failover#
const dispatcher = createJobDispatcher({
costEstimator,
providers: {
/* ... */
},
failoverConfig: {
enabled: true,
maxAttempts: 3, // Max failover attempts
providers: [
// Failover order
'runpod-serverless',
'modal',
'runpod-pod',
],
// Conditions for failover
failoverOn: [
'provider-unavailable',
'capacity-exhausted',
'timeout',
'rate-limited',
],
// Don't failover for these errors
noFailoverOn: ['invalid-input', 'budget-exceeded', 'authentication-failed'],
},
});
Manual Failover#
// Force job to specific provider
const result = await dispatcher.dispatch(job, {
forceProvider: 'modal',
});
// Retry on different provider
try {
await dispatcher.dispatch(job);
} catch (error) {
if (error.code === 'PROVIDER_UNAVAILABLE') {
await dispatcher.dispatch(job, {
excludeProviders: [error.provider],
});
}
}
Health-Based Routing#
const dispatcher = createJobDispatcher({
costEstimator,
providers: {
/* ... */
},
healthConfig: {
checkInterval: 30000, // Health check interval (ms)
unhealthyThreshold: 3, // Failures before unhealthy
healthyThreshold: 2, // Successes before healthy
// Route away from unhealthy providers
routeAroundUnhealthy: true,
},
});