# GPU Jobs Local Development Guide

This guide covers setting up and using the GPU Jobs system for local development
and testing.

## Table of Contents

- [Prerequisites](#prerequisites)
- [Quick Setup](#quick-setup)
- [Local GPU Setup](#local-gpu-setup)
- [Mock Provider](#mock-provider)
- [Testing Without GPU](#testing-without-gpu)
- [Development Workflow](#development-workflow)
- [Debugging](#debugging)

## Prerequisites

### System Requirements

- Node.js 18+ and pnpm
- Docker and Docker Compose (for local infrastructure)
- For GPU testing: NVIDIA GPU with CUDA 11.8+

### Optional: NVIDIA Setup (for local GPU testing)

```bash
# Check NVIDIA driver
nvidia-smi

# Check CUDA version
nvcc --version

# Install NVIDIA Container Toolkit (for Docker GPU support)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
```

## Quick Setup

### 1. Install Dependencies

```bash
# From repository root
pnpm install

# Build GPU Jobs package
pnpm --filter @yemaya/gpu-jobs build
```

### 2. Environment Configuration

```bash
# Copy example environment
cp .env.example .env
```

Add to `.env`:

```bash
# GPU Jobs Configuration
GPU_JOBS_ENV=development

# Provider API Keys (optional for local-only development)
RUNPOD_API_KEY=your_runpod_api_key
RUNPOD_ENDPOINT_ID=your_endpoint_id
MODAL_TOKEN=your_modal_token

# AWS (optional)
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1

# Local GPU Configuration
LOCAL_GPU_IDS=0
LOCAL_MAX_CONCURRENT=1
LOCAL_WORKER_SCRIPT=./scripts/gpu-worker.py
```

### 3. Start Infrastructure

```bash
# Start PostgreSQL, Redis, MinIO
docker compose up -d

# Verify services are running
docker compose ps
```

### 4. Run Database Migrations

```bash
pnpm --filter @yemaya/database db:migrate
pnpm --filter @yemaya/database db:generate
```

## Local GPU Setup

### Configuration

```typescript
import {
  createJobDispatcher,
  createGPUCostEstimator,
  createLocalExecutor,
} from '@yemaya/gpu-jobs';

const localExecutor = createLocalExecutor({
  // GPU configuration
  gpuIds: [0], // Available GPU indices
  maxConcurrent: 1, // Max concurrent jobs

  // Worker configuration
  workerScript: './scripts/gpu-worker.py',
  pythonPath: '/usr/bin/python3',

  // Resource limits
  maxMemoryPerJob: 16 * 1024, // 16GB max memory
  maxDiskPerJob: 50 * 1024, // 50GB max disk

  // Paths
  workDir: '/tmp/gpu-jobs',
  outputDir: './outputs',
});

const dispatcher = createJobDispatcher({
  costEstimator: createGPUCostEstimator(),
  defaultProvider: 'local',
  providers: {
    local: localExecutor,
  },
});
```

### GPU Detection

```typescript
// Auto-detect available GPUs
const gpus = await localExecutor.detectGPUs();
console.log('Available GPUs:', gpus);
// [
//   { id: 0, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 0 },
//   { id: 1, name: 'NVIDIA RTX 4090', memory: 24576, utilization: 15 },
// ]

// Check if specific GPU is available
const isAvailable = await localExecutor.isGPUAvailable(0);
console.log('GPU 0 available:', isAvailable);
```

### Worker Script Template

Create `scripts/gpu-worker.py`:

```python
#!/usr/bin/env python3
"""
Local GPU worker for GPU Jobs system.
Handles job execution on local NVIDIA GPUs.
"""

import json
import sys
import os
import torch
from pathlib import Path

def process_job(job_input: dict) -> dict:
    """Process a GPU job and return results."""
    job_type = job_input.get('type')

    if job_type == 'texture-upscale':
        return process_texture_upscale(job_input)
    elif job_type == 'mesh-processing':
        return process_mesh(job_input)
    elif job_type == 'custom':
        return process_custom(job_input)
    else:
        raise ValueError(f"Unknown job type: {job_type}")

def process_texture_upscale(job_input: dict) -> dict:
    """Upscale texture using Real-ESRGAN."""
    from realesrgan import RealESRGAN

    model = RealESRGAN(device='cuda')
    input_path = job_input['inputPath']
    output_path = job_input['outputPath']
    scale = job_input.get('scaleFactor', 4)

    result = model.upscale(input_path, scale)
    result.save(output_path)

    return {
        'outputPath': output_path,
        'originalSize': job_input.get('originalSize'),
        'upscaledSize': [s * scale for s in job_input.get('originalSize', [0, 0])],
    }

def process_mesh(job_input: dict) -> dict:
    """Process 3D mesh on GPU."""
    # Mesh processing implementation
    pass

def process_custom(job_input: dict) -> dict:
    """Custom job processing."""
    script = job_input.get('script')
    if script:
        exec(script)
    return {'status': 'completed'}

if __name__ == '__main__':
    # Read job input from stdin
    input_data = json.load(sys.stdin)

    # Set CUDA device
    gpu_id = input_data.get('gpuId', 0)
    os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)

    # Process job
    try:
        result = process_job(input_data)
        print(json.dumps({'success': True, 'result': result}))
    except Exception as e:
        print(json.dumps({'success': False, 'error': str(e)}))
        sys.exit(1)
```

## Mock Provider

For testing without a GPU or provider credentials, use the mock provider:

### Configuration

```typescript
import {
  createJobDispatcher,
  createGPUCostEstimator,
  createMockExecutor,
} from '@yemaya/gpu-jobs';

const mockExecutor = createMockExecutor({
  // Simulated behavior
  simulatedDuration: 5000, // 5 second jobs
  simulatedFailureRate: 0.1, // 10% random failures
  simulatedColdStart: 1000, // 1 second cold start

  // Mock responses
  mockResponses: {
    'texture-upscale': {
      outputPath: '/mock/output/texture.png',
      upscaledSize: [4096, 4096],
    },
    'gaussian-splatting': {
      modelPath: '/mock/output/splat.ply',
      pointCount: 1000000,
    },
  },
});

const dispatcher = createJobDispatcher({
  costEstimator: createGPUCostEstimator(),
  defaultProvider: 'mock',
  providers: {
    mock: mockExecutor,
  },
});
```

### Usage

```typescript
// Jobs execute with mock responses
const result = await dispatcher.dispatch({
  type: 'texture-upscale',
  priority: 'normal',
  gpuRequirements: { minVRAM: 8 },
  input: { imageUrl: 'test.png' },
  output: { destinationPath: '/outputs' },
});

console.log(result);
// {
//   jobId: 'gpu_mock_123',
//   provider: 'mock',
//   status: 'completed',
//   output: { outputPath: '/mock/output/texture.png', upscaledSize: [4096, 4096] }
// }
```

## Testing Without GPU

### Unit Tests

```typescript
// packages/gpu-jobs/src/__tests__/dispatcher.test.ts
import { describe, it, expect, vi } from 'vitest';
import { createJobDispatcher, createMockExecutor } from '../index';

describe('JobDispatcher', () => {
  it('should dispatch jobs to mock provider', async () => {
    const mockExecutor = createMockExecutor({
      simulatedDuration: 100,
    });

    const dispatcher = createJobDispatcher({
      costEstimator: createGPUCostEstimator(),
      providers: { mock: mockExecutor },
      defaultProvider: 'mock',
    });

    const result = await dispatcher.dispatch({
      type: 'texture-upscale',
      priority: 'normal',
      gpuRequirements: { minVRAM: 8 },
      input: {},
      output: { destinationPath: '/test' },
    });

    expect(result.jobId).toBeDefined();
    expect(result.provider).toBe('mock');
  });
});
```

### Run Tests

```bash
# Run all GPU Jobs tests
pnpm --filter @yemaya/gpu-jobs test

# Run with coverage
pnpm --filter @yemaya/gpu-jobs test:coverage

# Run in watch mode
pnpm --filter @yemaya/gpu-jobs test:watch
```

### Integration Tests with Docker

```yaml
# docker-compose.test.yml
version: '3.8'
services:
  gpu-jobs-test:
    build:
      context: .
      dockerfile: packages/gpu-jobs/Dockerfile.test
    environment:
      - DATABASE_URL=postgresql://test:test@postgres:5432/test
      - REDIS_URL=redis://redis:6379
    depends_on:
      - postgres
      - redis

  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: test

  redis:
    image: redis:7
```

```bash
# Run integration tests
docker compose -f docker-compose.test.yml up --abort-on-container-exit
```

## Development Workflow

### 1. Start Development Mode

```bash
# Terminal 1: Start infrastructure
docker compose up -d

# Terminal 2: Build and watch GPU Jobs package
pnpm --filter @yemaya/gpu-jobs dev

# Terminal 3: Run API with hot reload
pnpm --filter @yemaya/api dev
```

### 2. Test Job Submission

```typescript
// scripts/test-dispatch.ts
import {
  createJobDispatcher,
  createGPUCostEstimator,
  createLocalExecutor,
} from '@yemaya/gpu-jobs';

async function main() {
  const dispatcher = createJobDispatcher({
    costEstimator: createGPUCostEstimator(),
    defaultProvider: 'local',
    providers: {
      local: createLocalExecutor({
        gpuIds: [0],
        maxConcurrent: 1,
      }),
    },
  });

  // Test cost estimation
  const estimate = dispatcher.estimateCost({
    type: 'texture-upscale',
    priority: 'normal',
    gpuRequirements: { minVRAM: 8, estimatedDuration: 60 },
    input: { scaleFactor: 4 },
    output: { destinationPath: '/outputs' },
  });
  console.log('Estimated cost:', estimate);

  // Test job dispatch
  const result = await dispatcher.dispatch({
    type: 'texture-upscale',
    priority: 'normal',
    gpuRequirements: { minVRAM: 8 },
    input: {
      imageUrl: 'https://example.com/test.png',
      scaleFactor: 4,
    },
    output: {
      destinationPath: '/outputs',
    },
  });
  console.log('Job result:', result);

  // Check status
  const status = await dispatcher.getStatus(result.jobId);
  console.log('Job status:', status);
}

main().catch(console.error);
```

Run:

```bash
pnpm tsx scripts/test-dispatch.ts
```

### 3. API Testing

```bash
# Submit job via API
curl -X POST http://localhost:3000/api/v1/gpu-jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "type": "texture-upscale",
    "priority": "normal",
    "gpuRequirements": {
      "minVRAM": 8,
      "preferredGPU": "rtx-4090"
    },
    "input": {
      "imageUrl": "https://example.com/texture.png",
      "scaleFactor": 4
    },
    "output": {
      "destinationPath": "/textures/upscaled"
    }
  }'

# Check job status
curl http://localhost:3000/api/v1/gpu-jobs/gpu_123456 \
  -H "Authorization: Bearer $API_KEY"
```

## Debugging

### Enable Debug Logging

```typescript
import { setLogLevel } from '@yemaya/gpu-jobs';

// Enable all debug logs
setLogLevel('debug');

// Component-specific logging
setLogLevel('debug', 'dispatcher');
setLogLevel('debug', 'cost-estimator');
setLogLevel('debug', 'provider:local');
```

### VS Code Debug Configuration

Add to `.vscode/launch.json`:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug GPU Jobs Test",
      "runtimeExecutable": "pnpm",
      "runtimeArgs": ["--filter", "@yemaya/gpu-jobs", "test", "--", "--run"],
      "cwd": "${workspaceFolder}",
      "console": "integratedTerminal"
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Test Dispatch Script",
      "runtimeExecutable": "pnpm",
      "runtimeArgs": ["tsx", "scripts/test-dispatch.ts"],
      "cwd": "${workspaceFolder}",
      "console": "integratedTerminal",
      "env": {
        "DEBUG": "gpu-jobs:*"
      }
    }
  ]
}
```

### GPU Monitoring During Development

```bash
# Watch GPU usage in real-time
watch -n 1 nvidia-smi

# Or use nvtop for better visualization
nvtop
```

### Common Development Issues

#### "Module not found" errors

```bash
# Rebuild all packages
pnpm build

# Or rebuild specific package
pnpm --filter @yemaya/gpu-jobs build
```

#### Database connection errors

```bash
# Check PostgreSQL is running
docker compose ps postgres

# Check connection string
echo $DATABASE_URL

# Run migrations
pnpm --filter @yemaya/database db:migrate
```

#### GPU not detected

```bash
# Verify NVIDIA driver
nvidia-smi

# Check Docker GPU support
docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
```

## Related Documentation

- [GPU Jobs README](./README.md)
- [Provider Configuration](./providers.md)
- [Cost Model](./cost-model.md)
- [Troubleshooting](./troubleshooting.md)
