# GPU Jobs User Guide

This guide covers how to use the GPU Jobs system for submitting and managing GPU
compute jobs.

## Table of Contents

- [Getting Started](#getting-started)
- [Job Submission](#job-submission)
- [Job Types](#job-types)
- [Monitoring Jobs](#monitoring-jobs)
- [Webhooks](#webhooks)
- [Cost Management](#cost-management)
- [SDK Usage](#sdk-usage)
- [Best Practices](#best-practices)
- [Examples](#examples)

## Getting Started

### Authentication

All API requests require authentication via API key or JWT token.

```bash
# Using API key (in header)
curl -H "Authorization: Bearer YOUR_API_KEY" \
  https://api.yemaya.example.com/api/v1/gpu-jobs

# Using API key (as query parameter)
curl "https://api.yemaya.example.com/api/v1/gpu-jobs?api_key=YOUR_API_KEY"
```

### Base URL

```
Production: https://api.yemaya.example.com/api/v1
Staging: https://api-staging.yemaya.example.com/api/v1
```

### Rate Limits

| Tier       | Requests/Minute | Concurrent Jobs |
| ---------- | --------------- | --------------- |
| Free       | 10              | 2               |
| Pro        | 100             | 10              |
| Enterprise | 1000            | 100             |

Rate limit headers are included in all responses:

```
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1699920000
```

## Job Submission

### Submit a Job

```bash
POST /api/v1/gpu-jobs
Content-Type: application/json

{
  "type": "texture-upscale",
  "priority": "normal",
  "gpuRequirements": {
    "minVRAM": 8,
    "preferredGPU": "rtx-4090",
    "canUseSpot": true,
    "estimatedDuration": 120
  },
  "input": {
    "imageUrl": "https://storage.example.com/textures/input.png",
    "scaleFactor": 4,
    "algorithm": "real-esrgan-x4"
  },
  "output": {
    "destinationPath": "textures/upscaled/",
    "presignedUrls": true,
    "urlExpiration": 3600
  },
  "webhookUrl": "https://your-server.com/webhooks/gpu-jobs",
  "costConstraints": {
    "maxCost": 1.00
  }
}
```

### Response

```json
{
  "success": true,
  "data": {
    "jobId": "gpu_abc123_def456",
    "status": "pending",
    "provider": "runpod-serverless",
    "createdAt": "2024-01-15T10:30:00Z",
    "costEstimate": {
      "estimatedCost": 0.0534,
      "estimatedDuration": 63,
      "currency": "USD"
    }
  }
}
```

### Get Cost Estimate (Without Submitting)

```bash
POST /api/v1/gpu-jobs/estimate
Content-Type: application/json

{
  "type": "gaussian-splatting",
  "gpuRequirements": {
    "minVRAM": 40,
    "estimatedDuration": 3600
  },
  "input": {
    "pointCount": 1000000,
    "iterations": 30000
  }
}
```

### Response

```json
{
  "success": true,
  "data": {
    "estimates": [
      {
        "provider": "runpod-serverless",
        "gpuType": "a100-40gb",
        "estimatedCost": 5.0,
        "estimatedDuration": 3600,
        "breakdown": {
          "compute": 4.95,
          "transfer": 0.05,
          "platform": 0.0
        }
      },
      {
        "provider": "modal",
        "gpuType": "a100-40gb",
        "estimatedCost": 3.88,
        "estimatedDuration": 3600
      },
      {
        "provider": "aws-ecs-spot",
        "gpuType": "a100-40gb",
        "estimatedCost": 2.46,
        "estimatedDuration": 3750
      }
    ],
    "cheapest": {
      "provider": "aws-ecs-spot",
      "estimatedCost": 2.46
    },
    "fastest": {
      "provider": "modal",
      "estimatedDuration": 3600
    }
  }
}
```

## Job Types

### texture-upscale

AI-powered texture upscaling using Real-ESRGAN, SwinIR, or similar models.

```json
{
  "type": "texture-upscale",
  "input": {
    "imageUrl": "https://...",
    "scaleFactor": 4,
    "algorithm": "real-esrgan-x4",
    "denoiseStrength": 0.5,
    "faceEnhance": false
  },
  "gpuRequirements": {
    "minVRAM": 8
  }
}
```

| Parameter       | Type    | Required | Description                 |
| --------------- | ------- | -------- | --------------------------- |
| imageUrl        | string  | Yes      | URL of input image          |
| scaleFactor     | number  | Yes      | Upscale factor (2, 4, or 8) |
| algorithm       | string  | No       | Algorithm to use            |
| denoiseStrength | number  | No       | Denoise strength (0-1)      |
| faceEnhance     | boolean | No       | Enable face enhancement     |

### gaussian-splatting

3D Gaussian Splatting training from images.

```json
{
  "type": "gaussian-splatting",
  "input": {
    "imagesUrl": "https://storage.example.com/captures.zip",
    "colmapData": "https://storage.example.com/colmap.zip",
    "iterations": 30000,
    "shDegree": 3,
    "densifyInterval": 100
  },
  "gpuRequirements": {
    "minVRAM": 24
  }
}
```

| Parameter       | Type   | Required | Description                          |
| --------------- | ------ | -------- | ------------------------------------ |
| imagesUrl       | string | Yes      | URL to zipped images                 |
| colmapData      | string | No       | Pre-computed COLMAP data             |
| iterations      | number | No       | Training iterations (default: 30000) |
| shDegree        | number | No       | Spherical harmonics degree           |
| densifyInterval | number | No       | Densification interval               |

### blender-render

Blender Cycles or EEVEE rendering.

```json
{
  "type": "blender-render",
  "input": {
    "blendFileUrl": "https://storage.example.com/scene.blend",
    "frameStart": 1,
    "frameEnd": 250,
    "resolution": [1920, 1080],
    "engine": "cycles",
    "samples": 128,
    "denoising": true
  },
  "gpuRequirements": {
    "minVRAM": 16
  }
}
```

| Parameter    | Type    | Required | Description            |
| ------------ | ------- | -------- | ---------------------- |
| blendFileUrl | string  | Yes      | URL to .blend file     |
| frameStart   | number  | Yes      | Start frame            |
| frameEnd     | number  | Yes      | End frame              |
| resolution   | [w, h]  | No       | Output resolution      |
| engine       | string  | No       | "cycles" or "eevee"    |
| samples      | number  | No       | Render samples         |
| denoising    | boolean | No       | Enable OptiX denoising |

### mesh-processing

Mesh optimization, decimation, and unwrapping.

```json
{
  "type": "mesh-processing",
  "input": {
    "meshUrl": "https://storage.example.com/model.glb",
    "operations": ["decimate", "unwrap", "optimize"],
    "targetPolyCount": 50000,
    "uvMethod": "smart_project",
    "preserveUVs": true
  },
  "gpuRequirements": {
    "minVRAM": 8
  }
}
```

### video-generation

AI video generation using various models.

```json
{
  "type": "video-generation",
  "input": {
    "prompt": "A cinematic shot of a spaceship landing on Mars",
    "duration": 4,
    "fps": 24,
    "resolution": [1280, 720],
    "model": "runway-gen2"
  },
  "gpuRequirements": {
    "minVRAM": 40
  }
}
```

### custom

Custom GPU workloads with your own script.

```json
{
  "type": "custom",
  "input": {
    "scriptUrl": "https://storage.example.com/worker.py",
    "dataUrl": "https://storage.example.com/input.zip",
    "parameters": {
      "batchSize": 32,
      "epochs": 10
    }
  },
  "gpuRequirements": {
    "minVRAM": 24
  }
}
```

## Monitoring Jobs

### Get Job Status

```bash
GET /api/v1/gpu-jobs/{jobId}
```

```json
{
  "success": true,
  "data": {
    "jobId": "gpu_abc123_def456",
    "type": "texture-upscale",
    "status": "processing",
    "progress": {
      "percent": 65,
      "message": "Upscaling texture (step 2/3)",
      "currentStep": "upscale",
      "totalSteps": 3
    },
    "provider": "runpod-serverless",
    "gpuType": "rtx-4090",
    "createdAt": "2024-01-15T10:30:00Z",
    "startedAt": "2024-01-15T10:30:05Z",
    "costEstimate": {
      "estimatedCost": 0.0534,
      "currentCost": 0.0356
    }
  }
}
```

### Job Statuses

| Status         | Description                    |
| -------------- | ------------------------------ |
| `pending`      | Job queued, waiting for worker |
| `provisioning` | GPU worker starting            |
| `processing`   | Job actively running           |
| `uploading`    | Uploading results              |
| `completed`    | Job finished successfully      |
| `failed`       | Job failed with error          |
| `cancelled`    | Job cancelled by user          |
| `timeout`      | Job exceeded timeout           |

### List Jobs

```bash
GET /api/v1/gpu-jobs?status=completed&limit=20&offset=0
```

Query parameters:

- `status`: Filter by status (pending, processing, completed, failed)
- `type`: Filter by job type
- `limit`: Results per page (default: 20, max: 100)
- `offset`: Pagination offset
- `sort`: Sort field (createdAt, cost, duration)
- `order`: Sort order (asc, desc)

### Cancel Job

```bash
POST /api/v1/gpu-jobs/{jobId}/cancel
```

```json
{
  "success": true,
  "data": {
    "jobId": "gpu_abc123_def456",
    "status": "cancelled",
    "message": "Job cancelled successfully"
  }
}
```

## Webhooks

### Webhook Configuration

Add `webhookUrl` to your job submission to receive status updates:

```json
{
  "type": "texture-upscale",
  "webhookUrl": "https://your-server.com/webhooks/gpu-jobs",
  "webhookSecret": "your_webhook_secret",
  "webhookEvents": ["completed", "failed", "progress"]
}
```

### Webhook Events

#### Job Completed

```json
{
  "event": "job.completed",
  "timestamp": "2024-01-15T10:35:00Z",
  "data": {
    "jobId": "gpu_abc123_def456",
    "type": "texture-upscale",
    "status": "completed",
    "duration": 63,
    "cost": 0.0534,
    "output": {
      "files": [
        {
          "name": "output.png",
          "url": "https://storage.example.com/outputs/output.png",
          "size": 15728640,
          "expiresAt": "2024-01-15T11:35:00Z"
        }
      ]
    }
  },
  "signature": "sha256=..."
}
```

#### Job Failed

```json
{
  "event": "job.failed",
  "timestamp": "2024-01-15T10:35:00Z",
  "data": {
    "jobId": "gpu_abc123_def456",
    "type": "texture-upscale",
    "status": "failed",
    "error": {
      "code": "GPU_OUT_OF_MEMORY",
      "message": "CUDA out of memory. Required 12GB, available 8GB.",
      "retryable": true
    }
  },
  "signature": "sha256=..."
}
```

#### Job Progress

```json
{
  "event": "job.progress",
  "timestamp": "2024-01-15T10:32:00Z",
  "data": {
    "jobId": "gpu_abc123_def456",
    "progress": {
      "percent": 50,
      "message": "Upscaling texture...",
      "currentStep": 2,
      "totalSteps": 3
    }
  },
  "signature": "sha256=..."
}
```

### Webhook Signature Verification

```typescript
import crypto from 'crypto';

function verifyWebhook(
  payload: string,
  signature: string,
  secret: string
): boolean {
  const expectedSignature = `sha256=${crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex')}`;

  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  );
}

// Express.js example
app.post(
  '/webhooks/gpu-jobs',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signature = req.headers['x-webhook-signature'];
    const isValid = verifyWebhook(
      req.body.toString(),
      signature,
      WEBHOOK_SECRET
    );

    if (!isValid) {
      return res.status(401).json({ error: 'Invalid signature' });
    }

    const event = JSON.parse(req.body);
    // Handle event...

    res.status(200).json({ received: true });
  }
);
```

## Cost Management

### Get Budget Status

```bash
GET /api/v1/budget/status
```

```json
{
  "success": true,
  "data": {
    "daily": {
      "spent": 45.32,
      "limit": 100.0,
      "remaining": 54.68,
      "percentUsed": 45.32
    },
    "monthly": {
      "spent": 523.45,
      "limit": 1000.0,
      "remaining": 476.55,
      "percentUsed": 52.35
    },
    "alerts": [],
    "runningJobsCost": 3.21
  }
}
```

### Get Spending Report

```bash
GET /api/v1/budget/spending?period=week&groupBy=jobType
```

```json
{
  "success": true,
  "data": {
    "period": {
      "start": "2024-01-08T00:00:00Z",
      "end": "2024-01-15T00:00:00Z"
    },
    "totalSpent": 234.56,
    "breakdown": {
      "texture-upscale": 45.32,
      "gaussian-splatting": 123.45,
      "blender-render": 65.79
    },
    "jobs": {
      "total": 156,
      "completed": 148,
      "failed": 8
    }
  }
}
```

### Set Budget Alerts

```bash
POST /api/v1/budget/alerts
Content-Type: application/json

{
  "alerts": [
    {
      "type": "daily",
      "threshold": 80,
      "webhookUrl": "https://your-server.com/webhooks/budget"
    },
    {
      "type": "monthly",
      "threshold": 90,
      "webhookUrl": "https://your-server.com/webhooks/budget"
    }
  ]
}
```

## SDK Usage

### TypeScript/JavaScript

```typescript
import { YemayaClient } from '@yemaya/sdk';

const client = new YemayaClient({
  apiKey: process.env.YEMAYA_API_KEY,
  baseUrl: 'https://api.yemaya.example.com',
});

// Submit job
const job = await client.gpuJobs.create({
  type: 'texture-upscale',
  priority: 'normal',
  gpuRequirements: { minVRAM: 8 },
  input: {
    imageUrl: 'https://...',
    scaleFactor: 4,
  },
  output: {
    destinationPath: 'textures/',
    presignedUrls: true,
  },
});

console.log(`Job ID: ${job.jobId}`);
console.log(`Estimated cost: $${job.costEstimate.estimatedCost}`);

// Wait for completion
const result = await client.gpuJobs.waitForCompletion(job.jobId, {
  pollInterval: 5000,
  timeout: 300000,
});

console.log(`Output: ${result.output.files[0].url}`);
```

### React Hook

```tsx
import { useGPUJob } from '@yemaya/sdk/react';

function TextureUpscaler() {
  const { submit, status, progress, output, error, isLoading } = useGPUJob();

  const handleUpscale = async (imageUrl: string) => {
    await submit({
      type: 'texture-upscale',
      input: { imageUrl, scaleFactor: 4 },
      output: { destinationPath: 'upscaled/' },
    });
  };

  return (
    <div>
      {isLoading && <Progress value={progress?.percent} />}
      {error && <Error message={error.message} />}
      {output && <Image src={output.files[0].url} />}
      <Button onClick={() => handleUpscale(selectedImage)}>Upscale</Button>
    </div>
  );
}
```

### Python

```python
from yemaya import YemayaClient

client = YemayaClient(api_key="your_api_key")

# Submit job
job = client.gpu_jobs.create(
    type="texture-upscale",
    priority="normal",
    gpu_requirements={"min_vram": 8},
    input={"image_url": "https://...", "scale_factor": 4},
    output={"destination_path": "textures/"},
)

print(f"Job ID: {job.job_id}")

# Wait for completion
result = client.gpu_jobs.wait_for_completion(job.job_id)
print(f"Output: {result.output['files'][0]['url']}")
```

## Best Practices

### 1. Always Set Cost Constraints

```json
{
  "costConstraints": {
    "maxCost": 5.0
  }
}
```

This prevents unexpected charges if estimation is off.

### 2. Use Appropriate Priority

| Priority     | Use Case                       |
| ------------ | ------------------------------ |
| `urgent`     | User-facing, time-critical     |
| `high`       | Important but can wait briefly |
| `normal`     | Standard processing            |
| `low`        | Can wait for cheaper resources |
| `background` | Batch processing, no rush      |

### 3. Enable Spot Instances for Non-Critical Jobs

```json
{
  "gpuRequirements": {
    "canUseSpot": true
  }
}
```

Can reduce costs by 60-80%.

### 4. Provide Accurate Duration Estimates

```json
{
  "gpuRequirements": {
    "estimatedDuration": 300
  }
}
```

Helps with accurate cost estimation and scheduling.

### 5. Use Webhooks for Long-Running Jobs

Polling is inefficient. Use webhooks for jobs >30 seconds.

### 6. Batch Similar Jobs

Submit multiple similar jobs together for better resource utilization.

## Examples

### Example: Texture Upscaling Pipeline

```typescript
async function upscaleAllTextures(textureUrls: string[]) {
  const client = new YemayaClient({ apiKey: API_KEY });

  // Submit all jobs
  const jobs = await Promise.all(
    textureUrls.map((url) =>
      client.gpuJobs.create({
        type: 'texture-upscale',
        priority: 'normal',
        gpuRequirements: { minVRAM: 8, canUseSpot: true },
        input: { imageUrl: url, scaleFactor: 4 },
        output: { destinationPath: 'upscaled/', presignedUrls: true },
        costConstraints: { maxCost: 0.5 },
      })
    )
  );

  console.log(`Submitted ${jobs.length} jobs`);

  // Wait for all to complete
  const results = await Promise.all(
    jobs.map((job) => client.gpuJobs.waitForCompletion(job.jobId))
  );

  return results.map((r) => r.output.files[0].url);
}
```

### Example: 3D Gaussian Splatting Training

```typescript
async function trainGaussianSplat(imagesZipUrl: string) {
  const client = new YemayaClient({ apiKey: API_KEY });

  // Get cost estimate first
  const estimate = await client.gpuJobs.estimate({
    type: 'gaussian-splatting',
    gpuRequirements: { minVRAM: 40, estimatedDuration: 3600 },
    input: { imagesUrl: imagesZipUrl, iterations: 30000 },
  });

  console.log('Cost estimates:');
  estimate.estimates.forEach((e) => {
    console.log(`  ${e.provider}: $${e.estimatedCost.toFixed(2)}`);
  });

  // Use cheapest option
  const job = await client.gpuJobs.create({
    type: 'gaussian-splatting',
    priority: 'normal',
    gpuRequirements: {
      minVRAM: 40,
      canUseSpot: true, // Use spot for cost savings
    },
    input: {
      imagesUrl: imagesZipUrl,
      iterations: 30000,
      shDegree: 3,
    },
    output: {
      destinationPath: 'splats/',
      presignedUrls: true,
      urlExpiration: 86400, // 24 hours
    },
    webhookUrl: 'https://my-server.com/webhooks/gpu',
    costConstraints: {
      maxCost: estimate.cheapest.estimatedCost * 1.2, // 20% buffer
    },
  });

  return job.jobId;
}
```

### Example: Blender Animation Render

```typescript
async function renderAnimation(blendFileUrl: string, frames: number) {
  const client = new YemayaClient({ apiKey: API_KEY });

  // Split into chunks for parallel rendering
  const chunkSize = 50;
  const chunks: { start: number; end: number }[] = [];
  for (let i = 0; i < frames; i += chunkSize) {
    chunks.push({ start: i + 1, end: Math.min(i + chunkSize, frames) });
  }

  // Submit render jobs for each chunk
  const jobs = await Promise.all(
    chunks.map((chunk) =>
      client.gpuJobs.create({
        type: 'blender-render',
        priority: 'high',
        gpuRequirements: { minVRAM: 16, canUseSpot: false }, // No spot for reliability
        input: {
          blendFileUrl,
          frameStart: chunk.start,
          frameEnd: chunk.end,
          resolution: [1920, 1080],
          engine: 'cycles',
          samples: 128,
        },
        output: {
          destinationPath: `renders/frames_${chunk.start}_${chunk.end}/`,
          presignedUrls: true,
        },
      })
    )
  );

  console.log(`Submitted ${jobs.length} render chunks`);

  // Wait for all chunks
  const results = await Promise.all(
    jobs.map((job) => client.gpuJobs.waitForCompletion(job.jobId))
  );

  // Combine frame URLs
  const allFrameUrls = results.flatMap((r) => r.output.files.map((f) => f.url));
  return allFrameUrls.sort();
}
```

## Related Documentation

- [API Reference](./api.md)
- [Provider Configuration](./providers.md)
- [Cost Model](./cost-model.md)
- [Troubleshooting](./troubleshooting.md)
