# GPU Jobs Troubleshooting Guide

This guide covers common issues, debugging techniques, and solutions for the GPU
Jobs system.

## Table of Contents

- [Common Issues](#common-issues)
- [Provider-Specific Issues](#provider-specific-issues)
- [Cost & Budget Issues](#cost--budget-issues)
- [Performance Issues](#performance-issues)
- [Debug Mode](#debug-mode)
- [Health Checks](#health-checks)
- [Error Reference](#error-reference)

## Common Issues

### Job Stuck in "Provisioning" State

**Symptoms**: Job remains in `provisioning` status for longer than expected.

**Possible Causes**:

1. Provider capacity exhausted
2. Invalid API credentials
3. GPU type unavailable in region
4. Network connectivity issues

**Solutions**:

```typescript
// 1. Check provider health
const health = await dispatcher.healthCheck();
console.log('Provider status:', health.providers);

// 2. Force failover to another provider
await dispatcher.dispatch(job, {
  excludeProviders: ['runpod-serverless'],
});

// 3. Check capacity
const capacity = await dispatcher.getProviderCapacity('runpod-serverless');
console.log('Available:', capacity.available, '/', capacity.total);

// 4. Verify credentials
const isValid = await dispatcher.validateCredentials('runpod-serverless');
if (!isValid) {
  console.error('Invalid API credentials for RunPod');
}
```

### Job Failed with "GPU Out of Memory"

**Symptoms**: Job fails with OOM error or CUDA out of memory.

**Possible Causes**:

1. VRAM requirement underestimated
2. Batch size too large
3. Model too large for selected GPU

**Solutions**:

```typescript
// 1. Increase VRAM requirement
const job: GPUJobRequestBase = {
  gpuRequirements: {
    minVRAM: 24, // Increase from 16 to 24
    preferredGPU: 'a100-40gb', // Use GPU with more VRAM
  },
  // ...
};

// 2. Reduce batch size in job parameters
const job: GPUJobRequestBase = {
  input: {
    batchSize: 1, // Reduce from 4 to 1
    // ...
  },
  // ...
};

// 3. Use GPU with more VRAM
const estimate = costEstimator.getCheapestOption({
  ...job,
  gpuRequirements: { minVRAM: 48 },
});
```

### Job Timeout

**Symptoms**: Job exceeds timeout limit and is cancelled.

**Possible Causes**:

1. Duration estimate too low
2. GPU performance slower than expected
3. Large input data taking longer to process
4. Provider issues causing slowdown

**Solutions**:

```typescript
// 1. Increase timeout
const job: GPUJobRequestBase = {
  timeout: 7200, // Increase to 2 hours
  // ...
};

// 2. Use faster GPU
const job: GPUJobRequestBase = {
  gpuRequirements: {
    preferredGPU: 'h100', // Fastest available
  },
  // ...
};

// 3. Split job into smaller chunks
const chunks = splitJobInput(largeInput, { maxSize: 1000 });
const results = await Promise.all(
  chunks.map((chunk) => dispatcher.dispatch({ ...job, input: chunk }))
);
```

### Provider Unavailable

**Symptoms**: Dispatch fails with provider unavailable error.

**Possible Causes**:

1. Provider API down
2. Rate limiting
3. Account suspended
4. Network issues

**Solutions**:

```typescript
// 1. Check provider status
const status = await dispatcher.getProviderStatus('runpod-serverless');
console.log('Status:', status.health);
console.log('Last error:', status.lastError);

// 2. Use failover configuration
const dispatcher = createJobDispatcher({
  failoverConfig: {
    enabled: true,
    providers: ['runpod-serverless', 'modal', 'local'],
    maxAttempts: 3,
  },
});

// 3. Manual failover
try {
  await dispatcher.dispatch(job);
} catch (error) {
  if (error.code === 'PROVIDER_UNAVAILABLE') {
    await dispatcher.dispatch(job, {
      forceProvider: 'modal',
    });
  }
}
```

### Authentication Failed

**Symptoms**: API key rejected by provider.

**Possible Causes**:

1. Invalid or expired API key
2. Insufficient permissions
3. Wrong environment variable

**Solutions**:

```bash
# 1. Verify environment variables are set
echo $RUNPOD_API_KEY
echo $MODAL_TOKEN
echo $AWS_ACCESS_KEY_ID

# 2. Test API key directly
curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://api.runpod.ai/v2/endpoint/your-endpoint/health
```

```typescript
// 3. Validate credentials in code
const validation = await dispatcher.validateAllCredentials();
validation.forEach((result) => {
  console.log(`${result.provider}: ${result.valid ? 'OK' : result.error}`);
});
```

## Provider-Specific Issues

### RunPod Issues

#### Serverless Cold Start Too Long

```typescript
// Enable flash boot for faster cold starts
const executor = createRunPodServerlessExecutor({
  apiKey: process.env.RUNPOD_API_KEY!,
  endpointId: process.env.RUNPOD_ENDPOINT_ID!,
  workerConfig: {
    flashBoot: true,
    minWorkers: 1, // Keep at least 1 worker warm
  },
});
```

#### Pod Won't Start

```typescript
// Check available GPU types
const availability = await runpodExecutor.getGPUAvailability();
console.log('Available GPUs:', availability);

// Try different cloud type
const executor = createRunPodPodExecutor({
  podConfig: {
    cloudType: 'SECURE', // Try SECURE instead of COMMUNITY
  },
});
```

### Modal Issues

#### Function Not Found

```typescript
// Ensure Modal app is deployed
// Run: modal deploy your_app.py

// Check function exists
const functions = await modalExecutor.listFunctions();
console.log('Available functions:', functions);
```

#### Python Dependencies Missing

```typescript
// Add dependencies to image config
const executor = createModalExecutor({
  imageConfig: {
    pythonVersion: '3.10',
    pipPackages: [
      'torch',
      'transformers',
      'accelerate',
      'bitsandbytes', // Add missing package
    ],
  },
});
```

### AWS ECS Issues

#### Task Definition Not Found

```bash
# List available task definitions
aws ecs list-task-definitions --family-prefix gpu-worker

# Register new task definition
aws ecs register-task-definition --cli-input-json file://task-definition.json
```

#### No Container Instances

```typescript
// Check cluster capacity
const capacity = await ecsExecutor.getClusterCapacity();
console.log('Instances:', capacity.registeredContainerInstancesCount);
console.log('Running tasks:', capacity.runningTasksCount);

// Scale up capacity provider
await ecsExecutor.scaleCapacity({
  minCapacity: 1,
  maxCapacity: 10,
});
```

#### Spot Interruption

```typescript
// Handle spot interruptions
dispatcher.on('job-interrupted', async (event) => {
  if (event.reason === 'SPOT_INTERRUPTION') {
    // Resubmit on on-demand
    await dispatcher.dispatch(event.job, {
      forceProvider: 'aws-ecs-ondemand',
    });
  }
});
```

### Local Executor Issues

#### GPU Not Detected

```bash
# Check NVIDIA driver
nvidia-smi

# Check CUDA
nvcc --version

# Check PyTorch can see GPU
python -c "import torch; print(torch.cuda.is_available())"
```

```typescript
// List detected GPUs
const gpus = await localExecutor.detectGPUs();
if (gpus.length === 0) {
  console.error('No GPUs detected. Check NVIDIA drivers.');
}
```

#### Permission Denied

```bash
# Add user to docker group
sudo usermod -aG docker $USER

# Fix GPU device permissions
sudo chmod 666 /dev/nvidia*
```

## Cost & Budget Issues

### Estimated Cost Much Lower Than Actual

**Possible Causes**:

1. Large file transfers not accounted for
2. Cold start overhead
3. Job ran longer than estimated

**Solutions**:

```typescript
// 1. Include accurate data sizes
const job: GPUJobRequestBase = {
  input: {
    dataSize: 5 * 1024 * 1024 * 1024, // 5GB
  },
  // ...
};

// 2. Track estimate accuracy
dispatcher.on('job-completed', (event) => {
  const accuracy = event.estimatedCost / event.actualCost;
  console.log(`Estimate accuracy: ${(accuracy * 100).toFixed(1)}%`);

  if (accuracy < 0.8) {
    console.warn('Estimate was significantly off. Job details:', event);
  }
});
```

### Budget Exceeded Unexpectedly

```typescript
// 1. Get detailed budget breakdown
const status = await dispatcher.getBudgetStatus();
console.log('Daily breakdown:', status.daily);

// 2. Get spending by job type
const spending = await dispatcher.getSpendingByJobType({
  period: 'day',
});
console.log('Spending by type:', spending);

// 3. Identify expensive jobs
const expensiveJobs = await dispatcher.getRecentJobs({
  sortBy: 'cost',
  limit: 10,
});
console.log('Most expensive jobs:', expensiveJobs);
```

## Performance Issues

### Jobs Running Slower Than Expected

```typescript
// 1. Check GPU utilization
dispatcher.on('job-progress', (event) => {
  console.log('GPU utilization:', event.metrics?.gpuUtilization);
  console.log('Memory usage:', event.metrics?.memoryUsage);
});

// 2. Use faster GPU type
const job: GPUJobRequestBase = {
  gpuRequirements: {
    preferredGPU: 'h100', // Use H100 for maximum performance
  },
};

// 3. Enable multi-GPU
const job: GPUJobRequestBase = {
  gpuRequirements: {
    gpuCount: 2, // Use 2 GPUs
  },
};
```

### High Cold Start Latency

```typescript
// 1. Use warm pools
const warmPool = createWarmPoolManager({
  pools: {
    'texture-jobs': {
      minWorkers: 2,
      idleTimeout: 300,
    },
  },
});

// 2. Use providers with faster cold starts
// RunPod Serverless: ~3s
// Modal: ~10s
// AWS ECS: ~120s

const dispatcher = createJobDispatcher({
  selectionStrategy: 'fastest', // Prioritize fast cold starts
});
```

## Debug Mode

### Enable Debug Logging

```typescript
import { setLogLevel } from '@yemaya/gpu-jobs';

// Enable all debug logs
setLogLevel('debug');

// Enable for specific components
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');
setLogLevel('debug', 'provider:runpod');
setLogLevel('debug', 'state-manager');
```

### Log Output Example

```
[DEBUG] [dispatcher] Dispatching job gpu_1234 (texture-upscale)
[DEBUG] [cost-estimator] Estimating for providers: runpod-serverless, modal, local
[DEBUG] [cost-estimator] runpod-serverless/rtx-4090: $0.0413 (60s)
[DEBUG] [cost-estimator] modal/a10: $0.0306 (60s)
[DEBUG] [cost-estimator] local/rtx-4090: $0.0017 (60s)
[DEBUG] [dispatcher] Selected provider: local (cheapest)
[DEBUG] [provider:local] Executing job gpu_1234 on GPU 0
[DEBUG] [state-manager] Job gpu_1234 status: pending -> running
```

### Request/Response Logging

```typescript
// Log all provider API calls
const dispatcher = createJobDispatcher({
  debug: {
    logRequests: true,
    logResponses: true,
    logTiming: true,
  },
});
```

### Job Tracing

```typescript
// Enable distributed tracing
const dispatcher = createJobDispatcher({
  tracing: {
    enabled: true,
    serviceName: 'gpu-jobs',
    exporter: 'jaeger', // or 'zipkin', 'otlp'
    endpoint: 'http://localhost:14268/api/traces',
  },
});

// Each job gets a trace ID
const result = await dispatcher.dispatch(job);
console.log('Trace ID:', result.traceId);
```

## Health Checks

### System Health Check

```typescript
const health = await dispatcher.healthCheck();

console.log('System Health:', health.status);
console.log('Providers:');
health.providers.forEach((p) => {
  console.log(`  ${p.name}: ${p.status}`);
  if (p.status !== 'healthy') {
    console.log(`    Error: ${p.error}`);
    console.log(`    Last check: ${p.lastCheck}`);
  }
});
```

### Provider-Specific Health

```typescript
// Check single provider
const runpodHealth = await dispatcher.checkProviderHealth('runpod-serverless');
console.log('RunPod status:', runpodHealth);

// Get provider metrics
const metrics = await dispatcher.getProviderMetrics('runpod-serverless');
console.log('Metrics:', {
  successRate: metrics.successRate,
  avgLatency: metrics.avgLatency,
  errorRate: metrics.errorRate,
});
```

### Continuous Health Monitoring

```typescript
// Enable automatic health checks
const dispatcher = createJobDispatcher({
  healthConfig: {
    checkInterval: 30000, // Check every 30 seconds
    unhealthyThreshold: 3, // Mark unhealthy after 3 failures
    healthyThreshold: 2, // Mark healthy after 2 successes
    routeAroundUnhealthy: true, // Skip unhealthy providers
  },
});

// Listen for health changes
dispatcher.on('provider-unhealthy', (event) => {
  console.log(`Provider ${event.provider} is unhealthy: ${event.reason}`);
});

dispatcher.on('provider-healthy', (event) => {
  console.log(`Provider ${event.provider} recovered`);
});
```

## Error Reference

### Error Codes

| Code                      | Description                  | Resolution                                |
| ------------------------- | ---------------------------- | ----------------------------------------- |
| `PROVIDER_UNAVAILABLE`    | Provider API unreachable     | Check credentials, try different provider |
| `GPU_NOT_AVAILABLE`       | Requested GPU not available  | Try different GPU type or region          |
| `BUDGET_EXCEEDED`         | Job would exceed budget      | Increase budget or use cheaper options    |
| `INVALID_JOB_TYPE`        | Unsupported job type         | Check job type spelling                   |
| `VRAM_INSUFFICIENT`       | GPU doesn't have enough VRAM | Use GPU with more VRAM                    |
| `TIMEOUT`                 | Job exceeded timeout         | Increase timeout or optimize job          |
| `AUTHENTICATION_FAILED`   | Invalid credentials          | Check API keys                            |
| `RATE_LIMITED`            | Too many requests            | Add delay, reduce frequency               |
| `JOB_NOT_FOUND`           | Job ID doesn't exist         | Verify job ID                             |
| `PROVIDER_ERROR`          | Provider-specific error      | Check provider logs                       |
| `INPUT_VALIDATION_FAILED` | Invalid job input            | Check input parameters                    |
| `OUTPUT_UPLOAD_FAILED`    | Failed to upload results     | Check storage credentials                 |

### Error Handling Pattern

```typescript
import {
  GPUJobsError,
  ProviderError,
  BudgetError,
  TimeoutError,
} from '@yemaya/gpu-jobs';

try {
  await dispatcher.dispatch(job);
} catch (error) {
  if (error instanceof BudgetError) {
    console.log('Budget issue:', error.budgetType, error.currentSpend);
    // Handle budget exceeded
  } else if (error instanceof ProviderError) {
    console.log('Provider issue:', error.provider, error.code);
    // Try different provider
  } else if (error instanceof TimeoutError) {
    console.log('Timeout:', error.timeoutMs);
    // Retry with longer timeout
  } else if (error instanceof GPUJobsError) {
    console.log('GPU Jobs error:', error.code, error.message);
  } else {
    throw error; // Unknown error
  }
}
```

## Related Documentation

- [GPU Jobs README](./README.md)
- [Provider Configuration](./providers.md)
- [Cost Model](./cost-model.md)
- [API Reference](./api.md)
