Eve library

Video Processing Pipeline

Guide

Revision 1

On this page

This document provides comprehensive documentation for the Lilith Media Service Video Processing Pipeline, including video transcoding, HLS packaging for adaptive streaming, thumbnail generation, and animated previews.

Overview#

The Video Processing Pipeline provides:

  • Video Transcoding: Convert to H.264, H.265/HEVC, VP9, and AV1 codecs
  • HLS Packaging: Adaptive bitrate streaming with encryption support
  • Thumbnail Generation: Single frames, sprite grids, and WebVTT support
  • Animated Previews: GIF, WebP, and MP4 video previews

Table of Contents#

  1. Video Transcoding
  2. HLS Packaging
  3. Thumbnail Generation
  4. Animated Previews

Video Transcoding#

Supported Codecs#

Codec Encoder Container CRF Default Use Case
h264 libx264 .mp4 23 Broad compatibility, web streaming
h265 libx265 .mp4 28 Higher efficiency, modern devices
vp9 libvpx-vp9 .webm 31 Royalty-free, web (Chrome, Firefox)
av1 libaom-av1 .mp4 30 Best compression, future-proof

Hardware Acceleration#

Codec NVENC (NVIDIA) VAAPI (Linux) VideoToolbox (macOS) QSV (Intel)
H.264 h264_nvenc h264_vaapi h264_videotoolbox h264_qsv
H.265 hevc_nvenc hevc_vaapi hevc_videotoolbox hevc_qsv
VP9 - vp9_vaapi - vp9_qsv
AV1 av1_nvenc av1_vaapi - av1_qsv

Standard Resolutions#

Label Width Height Name
4k 3840 2160 4K UHD
1440p 2560 1440 1440p QHD
1080p 1920 1080 1080p Full HD
720p 1280 720 720p HD
480p 854 480 480p SD
360p 640 360 360p
240p 426 240 240p

Transcode Options#

typescript
interface VideoTranscodeOptions {
  codec: 'h264' | 'h265' | 'vp9' | 'av1';
  container?: 'mp4' | 'webm' | 'mkv';
  resolution?: VideoResolution | null;
  crf?: number; // Quality (lower = better, 18-28 typical)
  videoBitrate?: number; // kbps (alternative to CRF)
  maxBitrate?: number; // kbps
  bufferSize?: number; // kbps
  preset?: EncodingPreset; // Speed/quality tradeoff
  frameRate?: number | null;
  audioCodec?: 'aac' | 'opus' | 'mp3' | 'copy';
  audioBitrate?: number; // kbps
  audioSampleRate?: number;
  audioChannels?: number;
  twoPass?: boolean; // Better bitrate targeting
  pixelFormat?: string; // e.g., 'yuv420p'
  hwAccel?: 'nvenc' | 'vaapi' | 'videotoolbox' | 'qsv' | null;
  extraArgs?: string[];
  onProgress?: (progress: FFmpegProgress) => void;
  timeout?: number; // ms (default: 2 hours)
}

type EncodingPreset =
  | 'ultrafast'
  | 'superfast'
  | 'veryfast'
  | 'faster'
  | 'fast'
  | 'medium'
  | 'slow'
  | 'slower'
  | 'veryslow';

Transcoding Presets#

Preset Codec CRF Preset Audio Use Case
web_h264 H.264 23 medium AAC 128k Web streaming compatibility
web_h265 H.265 28 medium AAC 128k Modern devices, smaller files
web_vp9 VP9 31 medium Opus 128k Royalty-free WebM
fast_preview H.264 28 ultrafast AAC 96k Quick preview/draft
archive H.265 18 slow AAC 256k High-quality archival
mobile H.264 26 fast AAC 96k Mobile-optimized 720p
Resolution H.264 H.265/VP9 AV1
4K 35,000 kbps 21,000 kbps 17,500 kbps
1440p 16,000 kbps 9,600 kbps 8,000 kbps
1080p 8,000 kbps 4,800 kbps 4,000 kbps
720p 5,000 kbps 3,000 kbps 2,500 kbps
480p 2,500 kbps 1,500 kbps 1,250 kbps
360p 1,000 kbps 600 kbps 500 kbps

Usage Example#

typescript
import {
  transcodeVideo,
  transcodeToMultipleResolutions,
  getRecommendedBitrate,
  VIDEO_RESOLUTIONS,
  TRANSCODE_PRESETS,
} from './processing/video-transcoder.js';

// Single file transcoding
const result = await transcodeVideo(
  '/path/to/input.mov',
  '/path/to/output.mp4',
  {
    codec: 'h264',
    resolution: VIDEO_RESOLUTIONS['1080p'],
    crf: 23,
    preset: 'medium',
    audioCodec: 'aac',
    audioBitrate: 128,
    pixelFormat: 'yuv420p',
    onProgress: (progress) => {
      console.log(`Progress: ${progress.percent}%`);
    },
  }
);

console.log(`Transcoded to ${result.outputCodec}`);

// Using presets
const webResult = await transcodeVideo(
  '/path/to/input.mov',
  '/path/to/web.mp4',
  TRANSCODE_PRESETS.web_h264
);

// Two-pass encoding for target bitrate
const twoPassResult = await transcodeVideo(
  '/path/to/input.mov',
  '/path/to/output.mp4',
  {
    codec: 'h264',
    videoBitrate: 5000,
    maxBitrate: 6000,
    bufferSize: 10000,
    twoPass: true,
  }
);

// Batch transcode to multiple resolutions
const resolutions = [
  VIDEO_RESOLUTIONS['1080p'],
  VIDEO_RESOLUTIONS['720p'],
  VIDEO_RESOLUTIONS['480p'],
];

const results = await transcodeToMultipleResolutions(
  '/path/to/input.mov',
  '/path/to/output/',
  resolutions,
  { codec: 'h264', preset: 'fast' },
  (i, total, label, progress) => {
    console.log(`[${i}/${total}] ${label}: ${progress?.percent || 0}%`);
  }
);

// Get recommended bitrate
const bitrate = getRecommendedBitrate(VIDEO_RESOLUTIONS['1080p'], 'h264');
console.log(`Recommended bitrate: ${bitrate} kbps`);

HLS Packaging#

Overview#

HLS (HTTP Live Streaming) packaging creates adaptive bitrate streams with multiple quality variants. The player automatically switches quality based on network conditions.

HLS Variant Configuration#

typescript
interface HLSVariant {
  label: string; // e.g., '1080p'
  width: number;
  height: number;
  videoBitrate: number; // kbps
  audioBitrate: number; // kbps
  frameRate?: number;
}

HLS Package Options#

typescript
interface HLSPackageOptions {
  outputDir: string;
  variants: HLSVariant[];
  segmentDuration?: number; // seconds (default: 6)
  playlistType?: 'vod' | 'event'; // default: 'vod'
  codec?: 'h264' | 'h265'; // default: 'h264'
  preset?: 'ultrafast' | 'fast' | 'medium' | 'slow';
  encryption?: HLSEncryptionOptions | null;
  audioOnly?: boolean;
  masterPlaylist?: string; // default: 'master.m3u8'
  baseUrl?: string;
  onProgress?: (variant: string, progress: FFmpegProgress) => void;
  timeout?: number;
}

Variant Presets#

Standard (Up to 1080p)#

Label Resolution Video Bitrate Audio Bitrate
1080p 1920x1080 5,000 kbps 192 kbps
720p 1280x720 2,800 kbps 128 kbps
480p 854x480 1,400 kbps 128 kbps
360p 640x360 800 kbps 96 kbps

High Quality (Up to 4K)#

Label Resolution Video Bitrate Audio Bitrate
4k 3840x2160 15,000 kbps 256 kbps
1440p 2560x1440 10,000 kbps 192 kbps
1080p 1920x1080 5,000 kbps 192 kbps
720p 1280x720 2,800 kbps 128 kbps
480p 854x480 1,400 kbps 128 kbps

Mobile (Bandwidth Optimized)#

Label Resolution Video Bitrate Audio Bitrate
720p 1280x720 2,000 kbps 96 kbps
480p 854x480 1,000 kbps 64 kbps
360p 640x360 600 kbps 64 kbps
240p 426x240 300 kbps 48 kbps

HLS Encryption#

typescript
interface HLSEncryptionOptions {
  keyUri: string; // URL where key will be served
  key?: Buffer; // 16 bytes AES-128 (generated if not provided)
  iv?: Buffer; // 16 bytes IV (generated if not provided)
  keyRotation?: number; // Rotate key every N segments (0 = single key)
}

Usage Example#

typescript
import {
  packageHLS,
  packageHLSWithPreset,
  generateVariantsForResolution,
  generateEncryptionKey,
  validateHLSOutput,
  getHLSStats,
  HLS_VARIANT_PRESETS,
} from './processing/hls-packager.js';

// Package with standard preset
const result = await packageHLSWithPreset(
  '/path/to/input.mp4',
  '/path/to/hls/',
  'standard'
);

console.log(`Master playlist: ${result.masterPlaylist}`);
console.log(`Variants: ${result.variantPlaylists.size}`);
console.log(`Total segments: ${result.totalSegments}`);

// Package with encryption
const { key, iv } = generateEncryptionKey();

const encryptedResult = await packageHLS('/path/to/input.mp4', {
  outputDir: '/path/to/hls-encrypted/',
  variants: HLS_VARIANT_PRESETS.standard,
  encryption: {
    keyUri: 'https://keys.lilith.app/encryption.key',
    key,
    iv,
  },
  segmentDuration: 6,
  codec: 'h264',
  onProgress: (variant, progress) => {
    console.log(`${variant}: ${progress.percent}%`);
  },
});

// Save encryption key for later serving
console.log(`Key: ${encryptedResult.encryptionKey?.toString('hex')}`);
console.log(`IV: ${encryptedResult.encryptionIv?.toString('hex')}`);

// Generate variants based on input resolution
const variants = generateVariantsForResolution(1920, 1080, 'h264');

const customResult = await packageHLS('/path/to/input.mp4', {
  outputDir: '/path/to/hls-custom/',
  variants,
  masterPlaylist: 'index.m3u8',
  baseUrl: 'https://cdn.lilith.app/videos/123/',
});

// Validate HLS output
const validation = await validateHLSOutput(customResult.masterPlaylist);
if (!validation.valid) {
  console.error('HLS validation errors:', validation.errors);
}

// Get HLS statistics
const stats = await getHLSStats('/path/to/hls/');
console.log(`Total size: ${(stats.totalSize / 1024 / 1024).toFixed(2)} MB`);
console.log(`Segments: ${stats.segmentCount}`);
console.log(
  `Average segment: ${(stats.averageSegmentSize / 1024).toFixed(2)} KB`
);

HLS Output Structure#

text
hls/
├── master.m3u8          # Master playlist
├── 1080p.m3u8           # 1080p variant playlist
├── 1080p_000.ts         # 1080p segment 0
├── 1080p_001.ts         # 1080p segment 1
├── 720p.m3u8            # 720p variant playlist
├── 720p_000.ts
├── 720p_001.ts
├── 480p.m3u8            # 480p variant playlist
├── 480p_000.ts
├── 480p_001.ts
└── encryption.key       # Encryption key (if enabled)

Thumbnail Generation#

Single Thumbnail#

typescript
interface ThumbnailOptions {
  width?: number; // Auto-calculates height
  height?: number; // Auto-calculates width
  time?: number; // seconds (default: 10% into video)
  format?: 'jpg' | 'png' | 'webp'; // default: 'jpg'
  quality?: number; // 1-100 (default: 85)
}

Thumbnail Grid (Sprite)#

typescript
interface ThumbnailGridOptions {
  columns?: number; // default: 5
  rows?: number; // default: 5
  thumbnailWidth?: number; // default: 160
  thumbnailHeight?: number; // default: 90
  format?: 'jpg' | 'png' | 'webp';
  quality?: number;
  interval?: number; // seconds (auto-calculated if not specified)
}

Usage Example#

typescript
import {
  generateThumbnail,
  generateThumbnailsAtTimes,
  generateThumbnailGrid,
  extractPosterImage,
  generateHoverPreviewStrip,
  THUMBNAIL_DEFAULTS,
} from './processing/video-thumbnails.js';

// Generate single thumbnail
const thumb = await generateThumbnail(
  '/path/to/video.mp4',
  '/path/to/thumb.jpg',
  {
    width: 320,
    time: 30,
    format: 'jpg',
    quality: 85,
  }
);

console.log(
  `Thumbnail: ${thumb.width}x${thumb.height} at ${thumb.timePosition}s`
);

// Generate thumbnails at specific times
const times = [10, 30, 60, 120, 180];
const thumbnails = await generateThumbnailsAtTimes(
  '/path/to/video.mp4',
  '/path/to/thumbnails/',
  times,
  { width: 320, format: 'jpg' }
);

// Generate thumbnail sprite grid
const sprite = await generateThumbnailGrid(
  '/path/to/video.mp4',
  '/path/to/sprite.jpg',
  {
    columns: 10,
    rows: 10,
    thumbnailWidth: 160,
    thumbnailHeight: 90,
  }
);

console.log(`Sprite: ${sprite.width}x${sprite.height}`);
console.log(`Thumbnails: ${sprite.thumbnailCount}`);
console.log(`WebVTT: ${sprite.vttPath}`);

// Extract high-quality poster image
const poster = await extractPosterImage(
  '/path/to/video.mp4',
  '/path/to/poster.jpg',
  { time: 30 }
);

// Generate hover preview strip
const strip = await generateHoverPreviewStrip(
  '/path/to/video.mp4',
  '/path/to/preview-strip.jpg',
  {
    frames: 10,
    frameWidth: 160,
    frameHeight: 90,
  }
);

console.log(
  `Strip: ${strip.width}x${strip.height}, ${strip.frameCount} frames`
);

WebVTT for Video Seeking#

Generated sprite grids include WebVTT files for video player thumbnail seeking:

vtt
WEBVTT

00:00:06.000 --> 00:00:12.000
sprite.jpg#xywh=0,0,160,90

00:00:12.000 --> 00:00:18.000
sprite.jpg#xywh=160,0,160,90

00:00:18.000 --> 00:00:24.000
sprite.jpg#xywh=320,0,160,90

Animated Previews#

Preview Options#

typescript
interface PreviewOptions {
  duration?: number; // seconds (default: 5)
  startTime?: number; // seconds (default: 10% into video)
  width?: number;
  height?: number;
  frameRate?: number; // default: 10
  format?: 'gif' | 'webp' | 'mp4'; // default: 'gif'
  quality?: number;
  speed?: number; // 1 = normal, 2 = 2x speed (default: 1)
  multiSegment?: boolean; // Sample multiple parts of video
  segmentCount?: number; // Number of segments for multiSegment
}

Usage Example#

typescript
import {
  generatePreview,
  PREVIEW_DEFAULTS,
} from './processing/video-thumbnails.js';

// Generate GIF preview
const gifPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/preview.gif',
  {
    duration: 5,
    startTime: 30,
    width: 320,
    frameRate: 10,
    format: 'gif',
  }
);

console.log(
  `GIF preview: ${gifPreview.width}x${gifPreview.height}, ${gifPreview.duration}s`
);

// Generate WebP preview (better compression)
const webpPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/preview.webp',
  {
    duration: 5,
    width: 320,
    frameRate: 15,
    format: 'webp',
    quality: 80,
  }
);

// Generate MP4 preview (no audio)
const mp4Preview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/preview.mp4',
  {
    duration: 10,
    width: 640,
    frameRate: 24,
    format: 'mp4',
  }
);

// Generate sped-up preview
const fastPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/fast-preview.gif',
  {
    duration: 10,
    speed: 2, // 2x speed, showing 10s of content in 5s
    width: 320,
    format: 'gif',
  }
);

// Generate multi-segment preview (samples from different parts)
const multiPreview = await generatePreview(
  '/path/to/video.mp4',
  '/path/to/multi-preview.gif',
  {
    duration: 6,
    multiSegment: true,
    segmentCount: 3, // 3 segments of 2s each
    width: 320,
    format: 'gif',
  }
);

Processing Pipeline Integration#

Requesting Processing Jobs#

bash
# Request HLS packaging
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "hls_packaging",
    "config": {
      "variants": "standard",
      "encryption": {
        "keyUri": "https://keys.lilith.app/video123.key"
      }
    }
  }'

# Request thumbnail generation
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "thumbnail",
    "config": {
      "width": 320,
      "time": 30,
      "format": "jpg"
    }
  }'

# Request preview generation
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "preview",
    "config": {
      "duration": 5,
      "format": "gif",
      "width": 320
    }
  }'

SDK Integration#

TypeScript/JavaScript#

typescript
import { LilithClient } from '@lilith/sdk';

const client = new LilithClient({
  apiKey: process.env.LILITH_API_KEY,
});

// Request HLS packaging
const hlsJob = await client.media.requestProcessing('asset-video123', {
  type: 'hls_packaging',
  config: {
    variants: 'standard',
    encryption: {
      keyUri: 'https://keys.lilith.app/video123.key',
    },
  },
});

// Monitor job progress
const status = await client.media.getProcessingJob(hlsJob.jobId);
console.log(`Status: ${status.status}, Progress: ${status.progress}%`);

// Get HLS stream URL
const streamUrl = await client.media.getStreamUrl('asset-video123', {
  quality: 'auto',
});

// Get thumbnail
const thumbnail = await client.media.getThumbnail('asset-video123', {
  width: 320,
  time: 30,
});


Questions?#

  • Technical Support: api-support@lilith.app
  • Documentation: GitHub Issues