Eve library

GPU Jobs Troubleshooting Guide

Guide

Revision 1

On this page

This guide covers common issues, debugging techniques, and solutions for the GPU Jobs system.

Table of Contents#

Common Issues#

Job Stuck in "Provisioning" State#

Symptoms: Job remains in provisioning status for longer than expected.

Possible Causes:

  1. Provider capacity exhausted
  2. Invalid API credentials
  3. GPU type unavailable in region
  4. Network connectivity issues

Solutions:

typescript
// 1. Check provider health
const health = await dispatcher.healthCheck();
console.log('Provider status:', health.providers);

// 2. Force failover to another provider
await dispatcher.dispatch(job, {
  excludeProviders: ['runpod-serverless'],
});

// 3. Check capacity
const capacity = await dispatcher.getProviderCapacity('runpod-serverless');
console.log('Available:', capacity.available, '/', capacity.total);

// 4. Verify credentials
const isValid = await dispatcher.validateCredentials('runpod-serverless');
if (!isValid) {
  console.error('Invalid API credentials for RunPod');
}

Job Failed with "GPU Out of Memory"#

Symptoms: Job fails with OOM error or CUDA out of memory.

Possible Causes:

  1. VRAM requirement underestimated
  2. Batch size too large
  3. Model too large for selected GPU

Solutions:

typescript
// 1. Increase VRAM requirement
const job: GPUJobRequestBase = {
  gpuRequirements: {
    minVRAM: 24, // Increase from 16 to 24
    preferredGPU: 'a100-40gb', // Use GPU with more VRAM
  },
  // ...
};

// 2. Reduce batch size in job parameters
const job: GPUJobRequestBase = {
  input: {
    batchSize: 1, // Reduce from 4 to 1
    // ...
  },
  // ...
};

// 3. Use GPU with more VRAM
const estimate = costEstimator.getCheapestOption({
  ...job,
  gpuRequirements: { minVRAM: 48 },
});

Job Timeout#

Symptoms: Job exceeds timeout limit and is cancelled.

Possible Causes:

  1. Duration estimate too low
  2. GPU performance slower than expected
  3. Large input data taking longer to process
  4. Provider issues causing slowdown

Solutions:

typescript
// 1. Increase timeout
const job: GPUJobRequestBase = {
  timeout: 7200, // Increase to 2 hours
  // ...
};

// 2. Use faster GPU
const job: GPUJobRequestBase = {
  gpuRequirements: {
    preferredGPU: 'h100', // Fastest available
  },
  // ...
};

// 3. Split job into smaller chunks
const chunks = splitJobInput(largeInput, { maxSize: 1000 });
const results = await Promise.all(
  chunks.map((chunk) => dispatcher.dispatch({ ...job, input: chunk }))
);

Provider Unavailable#

Symptoms: Dispatch fails with provider unavailable error.

Possible Causes:

  1. Provider API down
  2. Rate limiting
  3. Account suspended
  4. Network issues

Solutions:

typescript
// 1. Check provider status
const status = await dispatcher.getProviderStatus('runpod-serverless');
console.log('Status:', status.health);
console.log('Last error:', status.lastError);

// 2. Use failover configuration
const dispatcher = createJobDispatcher({
  failoverConfig: {
    enabled: true,
    providers: ['runpod-serverless', 'modal', 'local'],
    maxAttempts: 3,
  },
});

// 3. Manual failover
try {
  await dispatcher.dispatch(job);
} catch (error) {
  if (error.code === 'PROVIDER_UNAVAILABLE') {
    await dispatcher.dispatch(job, {
      forceProvider: 'modal',
    });
  }
}

Authentication Failed#

Symptoms: API key rejected by provider.

Possible Causes:

  1. Invalid or expired API key
  2. Insufficient permissions
  3. Wrong environment variable

Solutions:

bash
# 1. Verify environment variables are set
echo $RUNPOD_API_KEY
echo $MODAL_TOKEN
echo $AWS_ACCESS_KEY_ID

# 2. Test API key directly
curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://api.runpod.ai/v2/endpoint/your-endpoint/health
typescript
// 3. Validate credentials in code
const validation = await dispatcher.validateAllCredentials();
validation.forEach((result) => {
  console.log(`${result.provider}: ${result.valid ? 'OK' : result.error}`);
});

Provider-Specific Issues#

RunPod Issues#

Serverless Cold Start Too Long#

typescript
// Enable flash boot for faster cold starts
const executor = createRunPodServerlessExecutor({
  apiKey: process.env.RUNPOD_API_KEY!,
  endpointId: process.env.RUNPOD_ENDPOINT_ID!,
  workerConfig: {
    flashBoot: true,
    minWorkers: 1, // Keep at least 1 worker warm
  },
});

Pod Won't Start#

typescript
// Check available GPU types
const availability = await runpodExecutor.getGPUAvailability();
console.log('Available GPUs:', availability);

// Try different cloud type
const executor = createRunPodPodExecutor({
  podConfig: {
    cloudType: 'SECURE', // Try SECURE instead of COMMUNITY
  },
});

Function Not Found#

typescript
// Ensure Modal app is deployed
// Run: modal deploy your_app.py

// Check function exists
const functions = await modalExecutor.listFunctions();
console.log('Available functions:', functions);

Python Dependencies Missing#

typescript
// Add dependencies to image config
const executor = createModalExecutor({
  imageConfig: {
    pythonVersion: '3.10',
    pipPackages: [
      'torch',
      'transformers',
      'accelerate',
      'bitsandbytes', // Add missing package
    ],
  },
});

AWS ECS Issues#

Task Definition Not Found#

bash
# List available task definitions
aws ecs list-task-definitions --family-prefix gpu-worker

# Register new task definition
aws ecs register-task-definition --cli-input-json file://task-definition.json

No Container Instances#

typescript
// Check cluster capacity
const capacity = await ecsExecutor.getClusterCapacity();
console.log('Instances:', capacity.registeredContainerInstancesCount);
console.log('Running tasks:', capacity.runningTasksCount);

// Scale up capacity provider
await ecsExecutor.scaleCapacity({
  minCapacity: 1,
  maxCapacity: 10,
});

Spot Interruption#

typescript
// Handle spot interruptions
dispatcher.on('job-interrupted', async (event) => {
  if (event.reason === 'SPOT_INTERRUPTION') {
    // Resubmit on on-demand
    await dispatcher.dispatch(event.job, {
      forceProvider: 'aws-ecs-ondemand',
    });
  }
});

Local Executor Issues#

GPU Not Detected#

bash
# Check NVIDIA driver
nvidia-smi

# Check CUDA
nvcc --version

# Check PyTorch can see GPU
python -c "import torch; print(torch.cuda.is_available())"
typescript
// List detected GPUs
const gpus = await localExecutor.detectGPUs();
if (gpus.length === 0) {
  console.error('No GPUs detected. Check NVIDIA drivers.');
}

Permission Denied#

bash
# Add user to docker group
sudo usermod -aG docker $USER

# Fix GPU device permissions
sudo chmod 666 /dev/nvidia*

Cost & Budget Issues#

Estimated Cost Much Lower Than Actual#

Possible Causes:

  1. Large file transfers not accounted for
  2. Cold start overhead
  3. Job ran longer than estimated

Solutions:

typescript
// 1. Include accurate data sizes
const job: GPUJobRequestBase = {
  input: {
    dataSize: 5 * 1024 * 1024 * 1024, // 5GB
  },
  // ...
};

// 2. Track estimate accuracy
dispatcher.on('job-completed', (event) => {
  const accuracy = event.estimatedCost / event.actualCost;
  console.log(`Estimate accuracy: ${(accuracy * 100).toFixed(1)}%`);

  if (accuracy < 0.8) {
    console.warn('Estimate was significantly off. Job details:', event);
  }
});

Budget Exceeded Unexpectedly#

typescript
// 1. Get detailed budget breakdown
const status = await dispatcher.getBudgetStatus();
console.log('Daily breakdown:', status.daily);

// 2. Get spending by job type
const spending = await dispatcher.getSpendingByJobType({
  period: 'day',
});
console.log('Spending by type:', spending);

// 3. Identify expensive jobs
const expensiveJobs = await dispatcher.getRecentJobs({
  sortBy: 'cost',
  limit: 10,
});
console.log('Most expensive jobs:', expensiveJobs);

Performance Issues#

Jobs Running Slower Than Expected#

typescript
// 1. Check GPU utilization
dispatcher.on('job-progress', (event) => {
  console.log('GPU utilization:', event.metrics?.gpuUtilization);
  console.log('Memory usage:', event.metrics?.memoryUsage);
});

// 2. Use faster GPU type
const job: GPUJobRequestBase = {
  gpuRequirements: {
    preferredGPU: 'h100', // Use H100 for maximum performance
  },
};

// 3. Enable multi-GPU
const job: GPUJobRequestBase = {
  gpuRequirements: {
    gpuCount: 2, // Use 2 GPUs
  },
};

High Cold Start Latency#

typescript
// 1. Use warm pools
const warmPool = createWarmPoolManager({
  pools: {
    'texture-jobs': {
      minWorkers: 2,
      idleTimeout: 300,
    },
  },
});

// 2. Use providers with faster cold starts
// RunPod Serverless: ~3s
// Modal: ~10s
// AWS ECS: ~120s

const dispatcher = createJobDispatcher({
  selectionStrategy: 'fastest', // Prioritize fast cold starts
});

Debug Mode#

Enable Debug Logging#

typescript
import { setLogLevel } from '@yemaya/gpu-jobs';

// Enable all debug logs
setLogLevel('debug');

// Enable for specific components
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');
setLogLevel('debug', 'provider:runpod');
setLogLevel('debug', 'state-manager');

Log Output Example#

text
[DEBUG] [dispatcher] Dispatching job gpu_1234 (texture-upscale)
[DEBUG] [cost-estimator] Estimating for providers: runpod-serverless, modal, local
[DEBUG] [cost-estimator] runpod-serverless/rtx-4090: $0.0413 (60s)
[DEBUG] [cost-estimator] modal/a10: $0.0306 (60s)
[DEBUG] [cost-estimator] local/rtx-4090: $0.0017 (60s)
[DEBUG] [dispatcher] Selected provider: local (cheapest)
[DEBUG] [provider:local] Executing job gpu_1234 on GPU 0
[DEBUG] [state-manager] Job gpu_1234 status: pending -> running

Request/Response Logging#

typescript
// Log all provider API calls
const dispatcher = createJobDispatcher({
  debug: {
    logRequests: true,
    logResponses: true,
    logTiming: true,
  },
});

Job Tracing#

typescript
// Enable distributed tracing
const dispatcher = createJobDispatcher({
  tracing: {
    enabled: true,
    serviceName: 'gpu-jobs',
    exporter: 'jaeger', // or 'zipkin', 'otlp'
    endpoint: 'http://localhost:14268/api/traces',
  },
});

// Each job gets a trace ID
const result = await dispatcher.dispatch(job);
console.log('Trace ID:', result.traceId);

Health Checks#

System Health Check#

typescript
const health = await dispatcher.healthCheck();

console.log('System Health:', health.status);
console.log('Providers:');
health.providers.forEach((p) => {
  console.log(`  ${p.name}: ${p.status}`);
  if (p.status !== 'healthy') {
    console.log(`    Error: ${p.error}`);
    console.log(`    Last check: ${p.lastCheck}`);
  }
});

Provider-Specific Health#

typescript
// Check single provider
const runpodHealth = await dispatcher.checkProviderHealth('runpod-serverless');
console.log('RunPod status:', runpodHealth);

// Get provider metrics
const metrics = await dispatcher.getProviderMetrics('runpod-serverless');
console.log('Metrics:', {
  successRate: metrics.successRate,
  avgLatency: metrics.avgLatency,
  errorRate: metrics.errorRate,
});

Continuous Health Monitoring#

typescript
// Enable automatic health checks
const dispatcher = createJobDispatcher({
  healthConfig: {
    checkInterval: 30000, // Check every 30 seconds
    unhealthyThreshold: 3, // Mark unhealthy after 3 failures
    healthyThreshold: 2, // Mark healthy after 2 successes
    routeAroundUnhealthy: true, // Skip unhealthy providers
  },
});

// Listen for health changes
dispatcher.on('provider-unhealthy', (event) => {
  console.log(`Provider ${event.provider} is unhealthy: ${event.reason}`);
});

dispatcher.on('provider-healthy', (event) => {
  console.log(`Provider ${event.provider} recovered`);
});

Error Reference#

Error Codes#

Code Description Resolution
PROVIDER_UNAVAILABLE Provider API unreachable Check credentials, try different provider
GPU_NOT_AVAILABLE Requested GPU not available Try different GPU type or region
BUDGET_EXCEEDED Job would exceed budget Increase budget or use cheaper options
INVALID_JOB_TYPE Unsupported job type Check job type spelling
VRAM_INSUFFICIENT GPU doesn't have enough VRAM Use GPU with more VRAM
TIMEOUT Job exceeded timeout Increase timeout or optimize job
AUTHENTICATION_FAILED Invalid credentials Check API keys
RATE_LIMITED Too many requests Add delay, reduce frequency
JOB_NOT_FOUND Job ID doesn't exist Verify job ID
PROVIDER_ERROR Provider-specific error Check provider logs
INPUT_VALIDATION_FAILED Invalid job input Check input parameters
OUTPUT_UPLOAD_FAILED Failed to upload results Check storage credentials

Error Handling Pattern#

typescript
import {
  GPUJobsError,
  ProviderError,
  BudgetError,
  TimeoutError,
} from '@yemaya/gpu-jobs';

try {
  await dispatcher.dispatch(job);
} catch (error) {
  if (error instanceof BudgetError) {
    console.log('Budget issue:', error.budgetType, error.currentSpend);
    // Handle budget exceeded
  } else if (error instanceof ProviderError) {
    console.log('Provider issue:', error.provider, error.code);
    // Try different provider
  } else if (error instanceof TimeoutError) {
    console.log('Timeout:', error.timeoutMs);
    // Retry with longer timeout
  } else if (error instanceof GPUJobsError) {
    console.log('GPU Jobs error:', error.code, error.message);
  } else {
    throw error; // Unknown error
  }
}