# Video Processing Pipeline

This document provides comprehensive documentation for the Lilith Media Service
Video Processing Pipeline, including video transcoding, HLS packaging for
adaptive streaming, thumbnail generation, and animated previews.

## Overview

The Video Processing Pipeline provides:

- **Video Transcoding**: Convert to H.264, H.265/HEVC, VP9, and AV1 codecs
- **HLS Packaging**: Adaptive bitrate streaming with encryption support
- **Thumbnail Generation**: Single frames, sprite grids, and WebVTT support
- **Animated Previews**: GIF, WebP, and MP4 video previews

---

## Table of Contents

1. [Video Transcoding](#video-transcoding)
2. [HLS Packaging](#hls-packaging)
3. [Thumbnail Generation](#thumbnail-generation)
4. [Animated Previews](#animated-previews)

---

## Video Transcoding

### Supported Codecs

| Codec  | Encoder    | Container | CRF Default | Use Case                            |
| ------ | ---------- | --------- | ----------- | ----------------------------------- |
| `h264` | libx264    | .mp4      | 23          | Broad compatibility, web streaming  |
| `h265` | libx265    | .mp4      | 28          | Higher efficiency, modern devices   |
| `vp9`  | libvpx-vp9 | .webm     | 31          | Royalty-free, web (Chrome, Firefox) |
| `av1`  | libaom-av1 | .mp4      | 30          | Best compression, future-proof      |

### Hardware Acceleration

| Codec | NVENC (NVIDIA) | VAAPI (Linux) | VideoToolbox (macOS) | QSV (Intel) |
| ----- | -------------- | ------------- | -------------------- | ----------- |
| H.264 | h264_nvenc     | h264_vaapi    | h264_videotoolbox    | h264_qsv    |
| H.265 | hevc_nvenc     | hevc_vaapi    | hevc_videotoolbox    | hevc_qsv    |
| VP9   | -              | vp9_vaapi     | -                    | vp9_qsv     |
| AV1   | av1_nvenc      | av1_vaapi     | -                    | av1_qsv     |

### Standard Resolutions

| Label   | Width | Height | Name          |
| ------- | ----- | ------ | ------------- |
| `4k`    | 3840  | 2160   | 4K UHD        |
| `1440p` | 2560  | 1440   | 1440p QHD     |
| `1080p` | 1920  | 1080   | 1080p Full HD |
| `720p`  | 1280  | 720    | 720p HD       |
| `480p`  | 854   | 480    | 480p SD       |
| `360p`  | 640   | 360    | 360p          |
| `240p`  | 426   | 240    | 240p          |

### Transcode Options

```typescript
interface VideoTranscodeOptions {
  codec: 'h264' | 'h265' | 'vp9' | 'av1';
  container?: 'mp4' | 'webm' | 'mkv';
  resolution?: VideoResolution | null;
  crf?: number; // Quality (lower = better, 18-28 typical)
  videoBitrate?: number; // kbps (alternative to CRF)
  maxBitrate?: number; // kbps
  bufferSize?: number; // kbps
  preset?: EncodingPreset; // Speed/quality tradeoff
  frameRate?: number | null;
  audioCodec?: 'aac' | 'opus' | 'mp3' | 'copy';
  audioBitrate?: number; // kbps
  audioSampleRate?: number;
  audioChannels?: number;
  twoPass?: boolean; // Better bitrate targeting
  pixelFormat?: string; // e.g., 'yuv420p'
  hwAccel?: 'nvenc' | 'vaapi' | 'videotoolbox' | 'qsv' | null;
  extraArgs?: string[];
  onProgress?: (progress: FFmpegProgress) => void;
  timeout?: number; // ms (default: 2 hours)
}

type EncodingPreset =
  | 'ultrafast'
  | 'superfast'
  | 'veryfast'
  | 'faster'
  | 'fast'
  | 'medium'
  | 'slow'
  | 'slower'
  | 'veryslow';
```

### Transcoding Presets

| Preset         | Codec | CRF | Preset    | Audio     | Use Case                      |
| -------------- | ----- | --- | --------- | --------- | ----------------------------- |
| `web_h264`     | H.264 | 23  | medium    | AAC 128k  | Web streaming compatibility   |
| `web_h265`     | H.265 | 28  | medium    | AAC 128k  | Modern devices, smaller files |
| `web_vp9`      | VP9   | 31  | medium    | Opus 128k | Royalty-free WebM             |
| `fast_preview` | H.264 | 28  | ultrafast | AAC 96k   | Quick preview/draft           |
| `archive`      | H.265 | 18  | slow      | AAC 256k  | High-quality archival         |
| `mobile`       | H.264 | 26  | fast      | AAC 96k   | Mobile-optimized 720p         |

### Recommended Bitrates

| Resolution | H.264       | H.265/VP9   | AV1         |
| ---------- | ----------- | ----------- | ----------- |
| 4K         | 35,000 kbps | 21,000 kbps | 17,500 kbps |
| 1440p      | 16,000 kbps | 9,600 kbps  | 8,000 kbps  |
| 1080p      | 8,000 kbps  | 4,800 kbps  | 4,000 kbps  |
| 720p       | 5,000 kbps  | 3,000 kbps  | 2,500 kbps  |
| 480p       | 2,500 kbps  | 1,500 kbps  | 1,250 kbps  |
| 360p       | 1,000 kbps  | 600 kbps    | 500 kbps    |

### Usage Example

```typescript
import {
  transcodeVideo,
  transcodeToMultipleResolutions,
  getRecommendedBitrate,
  VIDEO_RESOLUTIONS,
  TRANSCODE_PRESETS,
} from './processing/video-transcoder.js';

// Single file transcoding
const result = await transcodeVideo(
  '/path/to/input.mov',
  '/path/to/output.mp4',
  {
    codec: 'h264',
    resolution: VIDEO_RESOLUTIONS['1080p'],
    crf: 23,
    preset: 'medium',
    audioCodec: 'aac',
    audioBitrate: 128,
    pixelFormat: 'yuv420p',
    onProgress: (progress) => {
      console.log(`Progress: ${progress.percent}%`);
    },
  }
);

console.log(`Transcoded to ${result.outputCodec}`);

// Using presets
const webResult = await transcodeVideo(
  '/path/to/input.mov',
  '/path/to/web.mp4',
  TRANSCODE_PRESETS.web_h264
);

// Two-pass encoding for target bitrate
const twoPassResult = await transcodeVideo(
  '/path/to/input.mov',
  '/path/to/output.mp4',
  {
    codec: 'h264',
    videoBitrate: 5000,
    maxBitrate: 6000,
    bufferSize: 10000,
    twoPass: true,
  }
);

// Batch transcode to multiple resolutions
const resolutions = [
  VIDEO_RESOLUTIONS['1080p'],
  VIDEO_RESOLUTIONS['720p'],
  VIDEO_RESOLUTIONS['480p'],
];

const results = await transcodeToMultipleResolutions(
  '/path/to/input.mov',
  '/path/to/output/',
  resolutions,
  { codec: 'h264', preset: 'fast' },
  (i, total, label, progress) => {
    console.log(`[${i}/${total}] ${label}: ${progress?.percent || 0}%`);
  }
);

// Get recommended bitrate
const bitrate = getRecommendedBitrate(VIDEO_RESOLUTIONS['1080p'], 'h264');
console.log(`Recommended bitrate: ${bitrate} kbps`);
```

---

## HLS Packaging

### Overview

HLS (HTTP Live Streaming) packaging creates adaptive bitrate streams with
multiple quality variants. The player automatically switches quality based on
network conditions.

### HLS Variant Configuration

```typescript
interface HLSVariant {
  label: string; // e.g., '1080p'
  width: number;
  height: number;
  videoBitrate: number; // kbps
  audioBitrate: number; // kbps
  frameRate?: number;
}
```

### HLS Package Options

```typescript
interface HLSPackageOptions {
  outputDir: string;
  variants: HLSVariant[];
  segmentDuration?: number; // seconds (default: 6)
  playlistType?: 'vod' | 'event'; // default: 'vod'
  codec?: 'h264' | 'h265'; // default: 'h264'
  preset?: 'ultrafast' | 'fast' | 'medium' | 'slow';
  encryption?: HLSEncryptionOptions | null;
  audioOnly?: boolean;
  masterPlaylist?: string; // default: 'master.m3u8'
  baseUrl?: string;
  onProgress?: (variant: string, progress: FFmpegProgress) => void;
  timeout?: number;
}
```

### Variant Presets

#### Standard (Up to 1080p)

| Label | Resolution | Video Bitrate | Audio Bitrate |
| ----- | ---------- | ------------- | ------------- |
| 1080p | 1920x1080  | 5,000 kbps    | 192 kbps      |
| 720p  | 1280x720   | 2,800 kbps    | 128 kbps      |
| 480p  | 854x480    | 1,400 kbps    | 128 kbps      |
| 360p  | 640x360    | 800 kbps      | 96 kbps       |

#### High Quality (Up to 4K)

| Label | Resolution | Video Bitrate | Audio Bitrate |
| ----- | ---------- | ------------- | ------------- |
| 4k    | 3840x2160  | 15,000 kbps   | 256 kbps      |
| 1440p | 2560x1440  | 10,000 kbps   | 192 kbps      |
| 1080p | 1920x1080  | 5,000 kbps    | 192 kbps      |
| 720p  | 1280x720   | 2,800 kbps    | 128 kbps      |
| 480p  | 854x480    | 1,400 kbps    | 128 kbps      |

#### Mobile (Bandwidth Optimized)

| Label | Resolution | Video Bitrate | Audio Bitrate |
| ----- | ---------- | ------------- | ------------- |
| 720p  | 1280x720   | 2,000 kbps    | 96 kbps       |
| 480p  | 854x480    | 1,000 kbps    | 64 kbps       |
| 360p  | 640x360    | 600 kbps      | 64 kbps       |
| 240p  | 426x240    | 300 kbps      | 48 kbps       |

### HLS Encryption

```typescript
interface HLSEncryptionOptions {
  keyUri: string; // URL where key will be served
  key?: Buffer; // 16 bytes AES-128 (generated if not provided)
  iv?: Buffer; // 16 bytes IV (generated if not provided)
  keyRotation?: number; // Rotate key every N segments (0 = single key)
}
```

### Usage Example

```typescript
import {
  packageHLS,
  packageHLSWithPreset,
  generateVariantsForResolution,
  generateEncryptionKey,
  validateHLSOutput,
  getHLSStats,
  HLS_VARIANT_PRESETS,
} from './processing/hls-packager.js';

// Package with standard preset
const result = await packageHLSWithPreset(
  '/path/to/input.mp4',
  '/path/to/hls/',
  'standard'
);

console.log(`Master playlist: ${result.masterPlaylist}`);
console.log(`Variants: ${result.variantPlaylists.size}`);
console.log(`Total segments: ${result.totalSegments}`);

// Package with encryption
const { key, iv } = generateEncryptionKey();

const encryptedResult = await packageHLS('/path/to/input.mp4', {
  outputDir: '/path/to/hls-encrypted/',
  variants: HLS_VARIANT_PRESETS.standard,
  encryption: {
    keyUri: 'https://keys.lilith.app/encryption.key',
    key,
    iv,
  },
  segmentDuration: 6,
  codec: 'h264',
  onProgress: (variant, progress) => {
    console.log(`${variant}: ${progress.percent}%`);
  },
});

// Save encryption key for later serving
console.log(`Key: ${encryptedResult.encryptionKey?.toString('hex')}`);
console.log(`IV: ${encryptedResult.encryptionIv?.toString('hex')}`);

// Generate variants based on input resolution
const variants = generateVariantsForResolution(1920, 1080, 'h264');

const customResult = await packageHLS('/path/to/input.mp4', {
  outputDir: '/path/to/hls-custom/',
  variants,
  masterPlaylist: 'index.m3u8',
  baseUrl: 'https://cdn.lilith.app/videos/123/',
});

// Validate HLS output
const validation = await validateHLSOutput(customResult.masterPlaylist);
if (!validation.valid) {
  console.error('HLS validation errors:', validation.errors);
}

// Get HLS statistics
const stats = await getHLSStats('/path/to/hls/');
console.log(`Total size: ${(stats.totalSize / 1024 / 1024).toFixed(2)} MB`);
console.log(`Segments: ${stats.segmentCount}`);
console.log(
  `Average segment: ${(stats.averageSegmentSize / 1024).toFixed(2)} KB`
);
```

### HLS Output Structure

```
hls/
├── master.m3u8          # Master playlist
├── 1080p.m3u8           # 1080p variant playlist
├── 1080p_000.ts         # 1080p segment 0
├── 1080p_001.ts         # 1080p segment 1
├── 720p.m3u8            # 720p variant playlist
├── 720p_000.ts
├── 720p_001.ts
├── 480p.m3u8            # 480p variant playlist
├── 480p_000.ts
├── 480p_001.ts
└── encryption.key       # Encryption key (if enabled)
```

---

## Thumbnail Generation

### Single Thumbnail

```typescript
interface ThumbnailOptions {
  width?: number; // Auto-calculates height
  height?: number; // Auto-calculates width
  time?: number; // seconds (default: 10% into video)
  format?: 'jpg' | 'png' | 'webp'; // default: 'jpg'
  quality?: number; // 1-100 (default: 85)
}
```

### Thumbnail Grid (Sprite)

```typescript
interface ThumbnailGridOptions {
  columns?: number; // default: 5
  rows?: number; // default: 5
  thumbnailWidth?: number; // default: 160
  thumbnailHeight?: number; // default: 90
  format?: 'jpg' | 'png' | 'webp';
  quality?: number;
  interval?: number; // seconds (auto-calculated if not specified)
}
```

### Usage Example

```typescript
import {
  generateThumbnail,
  generateThumbnailsAtTimes,
  generateThumbnailGrid,
  extractPosterImage,
  generateHoverPreviewStrip,
  THUMBNAIL_DEFAULTS,
} from './processing/video-thumbnails.js';

// Generate single thumbnail
const thumb = await generateThumbnail(
  '/path/to/video.mp4',
  '/path/to/thumb.jpg',
  {
    width: 320,
    time: 30,
    format: 'jpg',
    quality: 85,
  }
);

console.log(
  `Thumbnail: ${thumb.width}x${thumb.height} at ${thumb.timePosition}s`
);

// Generate thumbnails at specific times
const times = [10, 30, 60, 120, 180];
const thumbnails = await generateThumbnailsAtTimes(
  '/path/to/video.mp4',
  '/path/to/thumbnails/',
  times,
  { width: 320, format: 'jpg' }
);

// Generate thumbnail sprite grid
const sprite = await generateThumbnailGrid(
  '/path/to/video.mp4',
  '/path/to/sprite.jpg',
  {
    columns: 10,
    rows: 10,
    thumbnailWidth: 160,
    thumbnailHeight: 90,
  }
);

console.log(`Sprite: ${sprite.width}x${sprite.height}`);
console.log(`Thumbnails: ${sprite.thumbnailCount}`);
console.log(`WebVTT: ${sprite.vttPath}`);

// Extract high-quality poster image
const poster = await extractPosterImage(
  '/path/to/video.mp4',
  '/path/to/poster.jpg',
  { time: 30 }
);

// Generate hover preview strip
const strip = await generateHoverPreviewStrip(
  '/path/to/video.mp4',
  '/path/to/preview-strip.jpg',
  {
    frames: 10,
    frameWidth: 160,
    frameHeight: 90,
  }
);

console.log(
  `Strip: ${strip.width}x${strip.height}, ${strip.frameCount} frames`
);
```

### WebVTT for Video Seeking

Generated sprite grids include WebVTT files for video player thumbnail seeking:

```vtt
WEBVTT

00:00:06.000 --> 00:00:12.000
sprite.jpg#xywh=0,0,160,90

00:00:12.000 --> 00:00:18.000
sprite.jpg#xywh=160,0,160,90

00:00:18.000 --> 00:00:24.000
sprite.jpg#xywh=320,0,160,90
```

---

## Animated Previews

### Preview Options

```typescript
interface PreviewOptions {
  duration?: number; // seconds (default: 5)
  startTime?: number; // seconds (default: 10% into video)
  width?: number;
  height?: number;
  frameRate?: number; // default: 10
  format?: 'gif' | 'webp' | 'mp4'; // default: 'gif'
  quality?: number;
  speed?: number; // 1 = normal, 2 = 2x speed (default: 1)
  multiSegment?: boolean; // Sample multiple parts of video
  segmentCount?: number; // Number of segments for multiSegment
}
```

### Usage Example

```typescript
import {
  generatePreview,
  PREVIEW_DEFAULTS,
} from './processing/video-thumbnails.js';

// Generate GIF preview
const gifPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/preview.gif',
  {
    duration: 5,
    startTime: 30,
    width: 320,
    frameRate: 10,
    format: 'gif',
  }
);

console.log(
  `GIF preview: ${gifPreview.width}x${gifPreview.height}, ${gifPreview.duration}s`
);

// Generate WebP preview (better compression)
const webpPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/preview.webp',
  {
    duration: 5,
    width: 320,
    frameRate: 15,
    format: 'webp',
    quality: 80,
  }
);

// Generate MP4 preview (no audio)
const mp4Preview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/preview.mp4',
  {
    duration: 10,
    width: 640,
    frameRate: 24,
    format: 'mp4',
  }
);

// Generate sped-up preview
const fastPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/fast-preview.gif',
  {
    duration: 10,
    speed: 2, // 2x speed, showing 10s of content in 5s
    width: 320,
    format: 'gif',
  }
);

// Generate multi-segment preview (samples from different parts)
const multiPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/multi-preview.gif',
  {
    duration: 6,
    multiSegment: true,
    segmentCount: 3, // 3 segments of 2s each
    width: 320,
    format: 'gif',
  }
);
```

---

## Processing Pipeline Integration

### Requesting Processing Jobs

```bash
# Request HLS packaging
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "hls_packaging",
    "config": {
      "variants": "standard",
      "encryption": {
        "keyUri": "https://keys.lilith.app/video123.key"
      }
    }
  }'

# Request thumbnail generation
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "thumbnail",
    "config": {
      "width": 320,
      "time": 30,
      "format": "jpg"
    }
  }'

# Request preview generation
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "preview",
    "config": {
      "duration": 5,
      "format": "gif",
      "width": 320
    }
  }'
```

---

## SDK Integration

### TypeScript/JavaScript

```typescript
import { LilithClient } from '@lilith/sdk';

const client = new LilithClient({
  apiKey: process.env.LILITH_API_KEY,
});

// Request HLS packaging
const hlsJob = await client.media.requestProcessing('asset-video123', {
  type: 'hls_packaging',
  config: {
    variants: 'standard',
    encryption: {
      keyUri: 'https://keys.lilith.app/video123.key',
    },
  },
});

// Monitor job progress
const status = await client.media.getProcessingJob(hlsJob.jobId);
console.log(`Status: ${status.status}, Progress: ${status.progress}%`);

// Get HLS stream URL
const streamUrl = await client.media.getStreamUrl('asset-video123', {
  quality: 'auto',
});

// Get thumbnail
const thumbnail = await client.media.getThumbnail('asset-video123', {
  width: 320,
  time: 30,
});
```

---

## Related Documentation

- [Asset Management API](./asset-api.md) - Asset upload and management
- [Audio Processing](./audio-processing.md) - Audio transcoding and
  normalization
- [Transcript API](./transcript-api.md) - Transcript management
- [CDN Setup](./cdn-setup.md) - CloudFront CDN configuration

---

## Questions?

- **Technical Support**: api-support@lilith.app
- **Documentation**: [GitHub Issues](https://github.com/lilith/platform/issues)
