This guide covers common issues, debugging techniques, and solutions for the GPU Jobs system.
Table of Contents#
- Common Issues
- Provider-Specific Issues
- Cost & Budget Issues
- Performance Issues
- Debug Mode
- Health Checks
- Error Reference
Common Issues#
Job Stuck in "Provisioning" State#
Symptoms: Job remains in provisioning status for longer than expected.
Possible Causes:
- Provider capacity exhausted
- Invalid API credentials
- GPU type unavailable in region
- Network connectivity issues
Solutions:
typescript
// 1. Check provider health
const health = await dispatcher.healthCheck();
console.log('Provider status:', health.providers);
// 2. Force failover to another provider
await dispatcher.dispatch(job, {
excludeProviders: ['runpod-serverless'],
});
// 3. Check capacity
const capacity = await dispatcher.getProviderCapacity('runpod-serverless');
console.log('Available:', capacity.available, '/', capacity.total);
// 4. Verify credentials
const isValid = await dispatcher.validateCredentials('runpod-serverless');
if (!isValid) {
console.error('Invalid API credentials for RunPod');
}
Job Failed with "GPU Out of Memory"#
Symptoms: Job fails with OOM error or CUDA out of memory.
Possible Causes:
- VRAM requirement underestimated
- Batch size too large
- Model too large for selected GPU
Solutions:
typescript
// 1. Increase VRAM requirement
const job: GPUJobRequestBase = {
gpuRequirements: {
minVRAM: 24, // Increase from 16 to 24
preferredGPU: 'a100-40gb', // Use GPU with more VRAM
},
// ...
};
// 2. Reduce batch size in job parameters
const job: GPUJobRequestBase = {
input: {
batchSize: 1, // Reduce from 4 to 1
// ...
},
// ...
};
// 3. Use GPU with more VRAM
const estimate = costEstimator.getCheapestOption({
...job,
gpuRequirements: { minVRAM: 48 },
});
Job Timeout#
Symptoms: Job exceeds timeout limit and is cancelled.
Possible Causes:
- Duration estimate too low
- GPU performance slower than expected
- Large input data taking longer to process
- Provider issues causing slowdown
Solutions:
typescript
// 1. Increase timeout
const job: GPUJobRequestBase = {
timeout: 7200, // Increase to 2 hours
// ...
};
// 2. Use faster GPU
const job: GPUJobRequestBase = {
gpuRequirements: {
preferredGPU: 'h100', // Fastest available
},
// ...
};
// 3. Split job into smaller chunks
const chunks = splitJobInput(largeInput, { maxSize: 1000 });
const results = await Promise.all(
chunks.map((chunk) => dispatcher.dispatch({ ...job, input: chunk }))
);
Provider Unavailable#
Symptoms: Dispatch fails with provider unavailable error.
Possible Causes:
- Provider API down
- Rate limiting
- Account suspended
- Network issues
Solutions:
typescript
// 1. Check provider status
const status = await dispatcher.getProviderStatus('runpod-serverless');
console.log('Status:', status.health);
console.log('Last error:', status.lastError);
// 2. Use failover configuration
const dispatcher = createJobDispatcher({
failoverConfig: {
enabled: true,
providers: ['runpod-serverless', 'modal', 'local'],
maxAttempts: 3,
},
});
// 3. Manual failover
try {
await dispatcher.dispatch(job);
} catch (error) {
if (error.code === 'PROVIDER_UNAVAILABLE') {
await dispatcher.dispatch(job, {
forceProvider: 'modal',
});
}
}
Authentication Failed#
Symptoms: API key rejected by provider.
Possible Causes:
- Invalid or expired API key
- Insufficient permissions
- Wrong environment variable
Solutions:
bash
# 1. Verify environment variables are set
echo $RUNPOD_API_KEY
echo $MODAL_TOKEN
echo $AWS_ACCESS_KEY_ID
# 2. Test API key directly
curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
https://api.runpod.ai/v2/endpoint/your-endpoint/health
typescript
// 3. Validate credentials in code
const validation = await dispatcher.validateAllCredentials();
validation.forEach((result) => {
console.log(`${result.provider}: ${result.valid ? 'OK' : result.error}`);
});
Provider-Specific Issues#
RunPod Issues#
Serverless Cold Start Too Long#
typescript
// Enable flash boot for faster cold starts
const executor = createRunPodServerlessExecutor({
apiKey: process.env.RUNPOD_API_KEY!,
endpointId: process.env.RUNPOD_ENDPOINT_ID!,
workerConfig: {
flashBoot: true,
minWorkers: 1, // Keep at least 1 worker warm
},
});
Pod Won't Start#
typescript
// Check available GPU types
const availability = await runpodExecutor.getGPUAvailability();
console.log('Available GPUs:', availability);
// Try different cloud type
const executor = createRunPodPodExecutor({
podConfig: {
cloudType: 'SECURE', // Try SECURE instead of COMMUNITY
},
});
Modal Issues#
Function Not Found#
typescript
// Ensure Modal app is deployed
// Run: modal deploy your_app.py
// Check function exists
const functions = await modalExecutor.listFunctions();
console.log('Available functions:', functions);
Python Dependencies Missing#
typescript
// Add dependencies to image config
const executor = createModalExecutor({
imageConfig: {
pythonVersion: '3.10',
pipPackages: [
'torch',
'transformers',
'accelerate',
'bitsandbytes', // Add missing package
],
},
});
AWS ECS Issues#
Task Definition Not Found#
bash
# List available task definitions
aws ecs list-task-definitions --family-prefix gpu-worker
# Register new task definition
aws ecs register-task-definition --cli-input-json file://task-definition.json
No Container Instances#
typescript
// Check cluster capacity
const capacity = await ecsExecutor.getClusterCapacity();
console.log('Instances:', capacity.registeredContainerInstancesCount);
console.log('Running tasks:', capacity.runningTasksCount);
// Scale up capacity provider
await ecsExecutor.scaleCapacity({
minCapacity: 1,
maxCapacity: 10,
});
Spot Interruption#
typescript
// Handle spot interruptions
dispatcher.on('job-interrupted', async (event) => {
if (event.reason === 'SPOT_INTERRUPTION') {
// Resubmit on on-demand
await dispatcher.dispatch(event.job, {
forceProvider: 'aws-ecs-ondemand',
});
}
});
Local Executor Issues#
GPU Not Detected#
bash
# Check NVIDIA driver
nvidia-smi
# Check CUDA
nvcc --version
# Check PyTorch can see GPU
python -c "import torch; print(torch.cuda.is_available())"
typescript
// List detected GPUs
const gpus = await localExecutor.detectGPUs();
if (gpus.length === 0) {
console.error('No GPUs detected. Check NVIDIA drivers.');
}
Permission Denied#
bash
# Add user to docker group
sudo usermod -aG docker $USER
# Fix GPU device permissions
sudo chmod 666 /dev/nvidia*
Cost & Budget Issues#
Estimated Cost Much Lower Than Actual#
Possible Causes:
- Large file transfers not accounted for
- Cold start overhead
- Job ran longer than estimated
Solutions:
typescript
// 1. Include accurate data sizes
const job: GPUJobRequestBase = {
input: {
dataSize: 5 * 1024 * 1024 * 1024, // 5GB
},
// ...
};
// 2. Track estimate accuracy
dispatcher.on('job-completed', (event) => {
const accuracy = event.estimatedCost / event.actualCost;
console.log(`Estimate accuracy: ${(accuracy * 100).toFixed(1)}%`);
if (accuracy < 0.8) {
console.warn('Estimate was significantly off. Job details:', event);
}
});
Budget Exceeded Unexpectedly#
typescript
// 1. Get detailed budget breakdown
const status = await dispatcher.getBudgetStatus();
console.log('Daily breakdown:', status.daily);
// 2. Get spending by job type
const spending = await dispatcher.getSpendingByJobType({
period: 'day',
});
console.log('Spending by type:', spending);
// 3. Identify expensive jobs
const expensiveJobs = await dispatcher.getRecentJobs({
sortBy: 'cost',
limit: 10,
});
console.log('Most expensive jobs:', expensiveJobs);
Performance Issues#
Jobs Running Slower Than Expected#
typescript
// 1. Check GPU utilization
dispatcher.on('job-progress', (event) => {
console.log('GPU utilization:', event.metrics?.gpuUtilization);
console.log('Memory usage:', event.metrics?.memoryUsage);
});
// 2. Use faster GPU type
const job: GPUJobRequestBase = {
gpuRequirements: {
preferredGPU: 'h100', // Use H100 for maximum performance
},
};
// 3. Enable multi-GPU
const job: GPUJobRequestBase = {
gpuRequirements: {
gpuCount: 2, // Use 2 GPUs
},
};
High Cold Start Latency#
typescript
// 1. Use warm pools
const warmPool = createWarmPoolManager({
pools: {
'texture-jobs': {
minWorkers: 2,
idleTimeout: 300,
},
},
});
// 2. Use providers with faster cold starts
// RunPod Serverless: ~3s
// Modal: ~10s
// AWS ECS: ~120s
const dispatcher = createJobDispatcher({
selectionStrategy: 'fastest', // Prioritize fast cold starts
});
Debug Mode#
Enable Debug Logging#
typescript
import { setLogLevel } from '@yemaya/gpu-jobs';
// Enable all debug logs
setLogLevel('debug');
// Enable for specific components
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');
setLogLevel('debug', 'provider:runpod');
setLogLevel('debug', 'state-manager');
Log Output Example#
text
[DEBUG] [dispatcher] Dispatching job gpu_1234 (texture-upscale)
[DEBUG] [cost-estimator] Estimating for providers: runpod-serverless, modal, local
[DEBUG] [cost-estimator] runpod-serverless/rtx-4090: $0.0413 (60s)
[DEBUG] [cost-estimator] modal/a10: $0.0306 (60s)
[DEBUG] [cost-estimator] local/rtx-4090: $0.0017 (60s)
[DEBUG] [dispatcher] Selected provider: local (cheapest)
[DEBUG] [provider:local] Executing job gpu_1234 on GPU 0
[DEBUG] [state-manager] Job gpu_1234 status: pending -> running
Request/Response Logging#
typescript
// Log all provider API calls
const dispatcher = createJobDispatcher({
debug: {
logRequests: true,
logResponses: true,
logTiming: true,
},
});
Job Tracing#
typescript
// Enable distributed tracing
const dispatcher = createJobDispatcher({
tracing: {
enabled: true,
serviceName: 'gpu-jobs',
exporter: 'jaeger', // or 'zipkin', 'otlp'
endpoint: 'http://localhost:14268/api/traces',
},
});
// Each job gets a trace ID
const result = await dispatcher.dispatch(job);
console.log('Trace ID:', result.traceId);
Health Checks#
System Health Check#
typescript
const health = await dispatcher.healthCheck();
console.log('System Health:', health.status);
console.log('Providers:');
health.providers.forEach((p) => {
console.log(` ${p.name}: ${p.status}`);
if (p.status !== 'healthy') {
console.log(` Error: ${p.error}`);
console.log(` Last check: ${p.lastCheck}`);
}
});
Provider-Specific Health#
typescript
// Check single provider
const runpodHealth = await dispatcher.checkProviderHealth('runpod-serverless');
console.log('RunPod status:', runpodHealth);
// Get provider metrics
const metrics = await dispatcher.getProviderMetrics('runpod-serverless');
console.log('Metrics:', {
successRate: metrics.successRate,
avgLatency: metrics.avgLatency,
errorRate: metrics.errorRate,
});
Continuous Health Monitoring#
typescript
// Enable automatic health checks
const dispatcher = createJobDispatcher({
healthConfig: {
checkInterval: 30000, // Check every 30 seconds
unhealthyThreshold: 3, // Mark unhealthy after 3 failures
healthyThreshold: 2, // Mark healthy after 2 successes
routeAroundUnhealthy: true, // Skip unhealthy providers
},
});
// Listen for health changes
dispatcher.on('provider-unhealthy', (event) => {
console.log(`Provider ${event.provider} is unhealthy: ${event.reason}`);
});
dispatcher.on('provider-healthy', (event) => {
console.log(`Provider ${event.provider} recovered`);
});
Error Reference#
Error Codes#
| Code | Description | Resolution |
|---|---|---|
PROVIDER_UNAVAILABLE |
Provider API unreachable | Check credentials, try different provider |
GPU_NOT_AVAILABLE |
Requested GPU not available | Try different GPU type or region |
BUDGET_EXCEEDED |
Job would exceed budget | Increase budget or use cheaper options |
INVALID_JOB_TYPE |
Unsupported job type | Check job type spelling |
VRAM_INSUFFICIENT |
GPU doesn't have enough VRAM | Use GPU with more VRAM |
TIMEOUT |
Job exceeded timeout | Increase timeout or optimize job |
AUTHENTICATION_FAILED |
Invalid credentials | Check API keys |
RATE_LIMITED |
Too many requests | Add delay, reduce frequency |
JOB_NOT_FOUND |
Job ID doesn't exist | Verify job ID |
PROVIDER_ERROR |
Provider-specific error | Check provider logs |
INPUT_VALIDATION_FAILED |
Invalid job input | Check input parameters |
OUTPUT_UPLOAD_FAILED |
Failed to upload results | Check storage credentials |
Error Handling Pattern#
typescript
import {
GPUJobsError,
ProviderError,
BudgetError,
TimeoutError,
} from '@yemaya/gpu-jobs';
try {
await dispatcher.dispatch(job);
} catch (error) {
if (error instanceof BudgetError) {
console.log('Budget issue:', error.budgetType, error.currentSpend);
// Handle budget exceeded
} else if (error instanceof ProviderError) {
console.log('Provider issue:', error.provider, error.code);
// Try different provider
} else if (error instanceof TimeoutError) {
console.log('Timeout:', error.timeoutMs);
// Retry with longer timeout
} else if (error instanceof GPUJobsError) {
console.log('GPU Jobs error:', error.code, error.message);
} else {
throw error; // Unknown error
}
}