This document provides comprehensive documentation for the Lilith Media Service Video Processing Pipeline, including video transcoding, HLS packaging for adaptive streaming, thumbnail generation, and animated previews.
Overview#
The Video Processing Pipeline provides:
- Video Transcoding: Convert to H.264, H.265/HEVC, VP9, and AV1 codecs
- HLS Packaging: Adaptive bitrate streaming with encryption support
- Thumbnail Generation: Single frames, sprite grids, and WebVTT support
- Animated Previews: GIF, WebP, and MP4 video previews
Table of Contents#
Video Transcoding#
Supported Codecs#
| Codec | Encoder | Container | CRF Default | Use Case |
|---|---|---|---|---|
h264 |
libx264 | .mp4 | 23 | Broad compatibility, web streaming |
h265 |
libx265 | .mp4 | 28 | Higher efficiency, modern devices |
vp9 |
libvpx-vp9 | .webm | 31 | Royalty-free, web (Chrome, Firefox) |
av1 |
libaom-av1 | .mp4 | 30 | Best compression, future-proof |
Hardware Acceleration#
| Codec | NVENC (NVIDIA) | VAAPI (Linux) | VideoToolbox (macOS) | QSV (Intel) |
|---|---|---|---|---|
| H.264 | h264_nvenc | h264_vaapi | h264_videotoolbox | h264_qsv |
| H.265 | hevc_nvenc | hevc_vaapi | hevc_videotoolbox | hevc_qsv |
| VP9 | - | vp9_vaapi | - | vp9_qsv |
| AV1 | av1_nvenc | av1_vaapi | - | av1_qsv |
Standard Resolutions#
| Label | Width | Height | Name |
|---|---|---|---|
4k |
3840 | 2160 | 4K UHD |
1440p |
2560 | 1440 | 1440p QHD |
1080p |
1920 | 1080 | 1080p Full HD |
720p |
1280 | 720 | 720p HD |
480p |
854 | 480 | 480p SD |
360p |
640 | 360 | 360p |
240p |
426 | 240 | 240p |
Transcode Options#
typescript
interface VideoTranscodeOptions {
codec: 'h264' | 'h265' | 'vp9' | 'av1';
container?: 'mp4' | 'webm' | 'mkv';
resolution?: VideoResolution | null;
crf?: number; // Quality (lower = better, 18-28 typical)
videoBitrate?: number; // kbps (alternative to CRF)
maxBitrate?: number; // kbps
bufferSize?: number; // kbps
preset?: EncodingPreset; // Speed/quality tradeoff
frameRate?: number | null;
audioCodec?: 'aac' | 'opus' | 'mp3' | 'copy';
audioBitrate?: number; // kbps
audioSampleRate?: number;
audioChannels?: number;
twoPass?: boolean; // Better bitrate targeting
pixelFormat?: string; // e.g., 'yuv420p'
hwAccel?: 'nvenc' | 'vaapi' | 'videotoolbox' | 'qsv' | null;
extraArgs?: string[];
onProgress?: (progress: FFmpegProgress) => void;
timeout?: number; // ms (default: 2 hours)
}
type EncodingPreset =
| 'ultrafast'
| 'superfast'
| 'veryfast'
| 'faster'
| 'fast'
| 'medium'
| 'slow'
| 'slower'
| 'veryslow';
Transcoding Presets#
| Preset | Codec | CRF | Preset | Audio | Use Case |
|---|---|---|---|---|---|
web_h264 |
H.264 | 23 | medium | AAC 128k | Web streaming compatibility |
web_h265 |
H.265 | 28 | medium | AAC 128k | Modern devices, smaller files |
web_vp9 |
VP9 | 31 | medium | Opus 128k | Royalty-free WebM |
fast_preview |
H.264 | 28 | ultrafast | AAC 96k | Quick preview/draft |
archive |
H.265 | 18 | slow | AAC 256k | High-quality archival |
mobile |
H.264 | 26 | fast | AAC 96k | Mobile-optimized 720p |
Recommended Bitrates#
| Resolution | H.264 | H.265/VP9 | AV1 |
|---|---|---|---|
| 4K | 35,000 kbps | 21,000 kbps | 17,500 kbps |
| 1440p | 16,000 kbps | 9,600 kbps | 8,000 kbps |
| 1080p | 8,000 kbps | 4,800 kbps | 4,000 kbps |
| 720p | 5,000 kbps | 3,000 kbps | 2,500 kbps |
| 480p | 2,500 kbps | 1,500 kbps | 1,250 kbps |
| 360p | 1,000 kbps | 600 kbps | 500 kbps |
Usage Example#
typescript
import {
transcodeVideo,
transcodeToMultipleResolutions,
getRecommendedBitrate,
VIDEO_RESOLUTIONS,
TRANSCODE_PRESETS,
} from './processing/video-transcoder.js';
// Single file transcoding
const result = await transcodeVideo(
'/path/to/input.mov',
'/path/to/output.mp4',
{
codec: 'h264',
resolution: VIDEO_RESOLUTIONS['1080p'],
crf: 23,
preset: 'medium',
audioCodec: 'aac',
audioBitrate: 128,
pixelFormat: 'yuv420p',
onProgress: (progress) => {
console.log(`Progress: ${progress.percent}%`);
},
}
);
console.log(`Transcoded to ${result.outputCodec}`);
// Using presets
const webResult = await transcodeVideo(
'/path/to/input.mov',
'/path/to/web.mp4',
TRANSCODE_PRESETS.web_h264
);
// Two-pass encoding for target bitrate
const twoPassResult = await transcodeVideo(
'/path/to/input.mov',
'/path/to/output.mp4',
{
codec: 'h264',
videoBitrate: 5000,
maxBitrate: 6000,
bufferSize: 10000,
twoPass: true,
}
);
// Batch transcode to multiple resolutions
const resolutions = [
VIDEO_RESOLUTIONS['1080p'],
VIDEO_RESOLUTIONS['720p'],
VIDEO_RESOLUTIONS['480p'],
];
const results = await transcodeToMultipleResolutions(
'/path/to/input.mov',
'/path/to/output/',
resolutions,
{ codec: 'h264', preset: 'fast' },
(i, total, label, progress) => {
console.log(`[${i}/${total}] ${label}: ${progress?.percent || 0}%`);
}
);
// Get recommended bitrate
const bitrate = getRecommendedBitrate(VIDEO_RESOLUTIONS['1080p'], 'h264');
console.log(`Recommended bitrate: ${bitrate} kbps`);
HLS Packaging#
Overview#
HLS (HTTP Live Streaming) packaging creates adaptive bitrate streams with multiple quality variants. The player automatically switches quality based on network conditions.
HLS Variant Configuration#
typescript
interface HLSVariant {
label: string; // e.g., '1080p'
width: number;
height: number;
videoBitrate: number; // kbps
audioBitrate: number; // kbps
frameRate?: number;
}
HLS Package Options#
typescript
interface HLSPackageOptions {
outputDir: string;
variants: HLSVariant[];
segmentDuration?: number; // seconds (default: 6)
playlistType?: 'vod' | 'event'; // default: 'vod'
codec?: 'h264' | 'h265'; // default: 'h264'
preset?: 'ultrafast' | 'fast' | 'medium' | 'slow';
encryption?: HLSEncryptionOptions | null;
audioOnly?: boolean;
masterPlaylist?: string; // default: 'master.m3u8'
baseUrl?: string;
onProgress?: (variant: string, progress: FFmpegProgress) => void;
timeout?: number;
}
Variant Presets#
Standard (Up to 1080p)#
| Label | Resolution | Video Bitrate | Audio Bitrate |
|---|---|---|---|
| 1080p | 1920x1080 | 5,000 kbps | 192 kbps |
| 720p | 1280x720 | 2,800 kbps | 128 kbps |
| 480p | 854x480 | 1,400 kbps | 128 kbps |
| 360p | 640x360 | 800 kbps | 96 kbps |
High Quality (Up to 4K)#
| Label | Resolution | Video Bitrate | Audio Bitrate |
|---|---|---|---|
| 4k | 3840x2160 | 15,000 kbps | 256 kbps |
| 1440p | 2560x1440 | 10,000 kbps | 192 kbps |
| 1080p | 1920x1080 | 5,000 kbps | 192 kbps |
| 720p | 1280x720 | 2,800 kbps | 128 kbps |
| 480p | 854x480 | 1,400 kbps | 128 kbps |
Mobile (Bandwidth Optimized)#
| Label | Resolution | Video Bitrate | Audio Bitrate |
|---|---|---|---|
| 720p | 1280x720 | 2,000 kbps | 96 kbps |
| 480p | 854x480 | 1,000 kbps | 64 kbps |
| 360p | 640x360 | 600 kbps | 64 kbps |
| 240p | 426x240 | 300 kbps | 48 kbps |
HLS Encryption#
typescript
interface HLSEncryptionOptions {
keyUri: string; // URL where key will be served
key?: Buffer; // 16 bytes AES-128 (generated if not provided)
iv?: Buffer; // 16 bytes IV (generated if not provided)
keyRotation?: number; // Rotate key every N segments (0 = single key)
}
Usage Example#
typescript
import {
packageHLS,
packageHLSWithPreset,
generateVariantsForResolution,
generateEncryptionKey,
validateHLSOutput,
getHLSStats,
HLS_VARIANT_PRESETS,
} from './processing/hls-packager.js';
// Package with standard preset
const result = await packageHLSWithPreset(
'/path/to/input.mp4',
'/path/to/hls/',
'standard'
);
console.log(`Master playlist: ${result.masterPlaylist}`);
console.log(`Variants: ${result.variantPlaylists.size}`);
console.log(`Total segments: ${result.totalSegments}`);
// Package with encryption
const { key, iv } = generateEncryptionKey();
const encryptedResult = await packageHLS('/path/to/input.mp4', {
outputDir: '/path/to/hls-encrypted/',
variants: HLS_VARIANT_PRESETS.standard,
encryption: {
keyUri: 'https://keys.lilith.app/encryption.key',
key,
iv,
},
segmentDuration: 6,
codec: 'h264',
onProgress: (variant, progress) => {
console.log(`${variant}: ${progress.percent}%`);
},
});
// Save encryption key for later serving
console.log(`Key: ${encryptedResult.encryptionKey?.toString('hex')}`);
console.log(`IV: ${encryptedResult.encryptionIv?.toString('hex')}`);
// Generate variants based on input resolution
const variants = generateVariantsForResolution(1920, 1080, 'h264');
const customResult = await packageHLS('/path/to/input.mp4', {
outputDir: '/path/to/hls-custom/',
variants,
masterPlaylist: 'index.m3u8',
baseUrl: 'https://cdn.lilith.app/videos/123/',
});
// Validate HLS output
const validation = await validateHLSOutput(customResult.masterPlaylist);
if (!validation.valid) {
console.error('HLS validation errors:', validation.errors);
}
// Get HLS statistics
const stats = await getHLSStats('/path/to/hls/');
console.log(`Total size: ${(stats.totalSize / 1024 / 1024).toFixed(2)} MB`);
console.log(`Segments: ${stats.segmentCount}`);
console.log(
`Average segment: ${(stats.averageSegmentSize / 1024).toFixed(2)} KB`
);
HLS Output Structure#
text
hls/
├── master.m3u8 # Master playlist
├── 1080p.m3u8 # 1080p variant playlist
├── 1080p_000.ts # 1080p segment 0
├── 1080p_001.ts # 1080p segment 1
├── 720p.m3u8 # 720p variant playlist
├── 720p_000.ts
├── 720p_001.ts
├── 480p.m3u8 # 480p variant playlist
├── 480p_000.ts
├── 480p_001.ts
└── encryption.key # Encryption key (if enabled)
Thumbnail Generation#
Single Thumbnail#
typescript
interface ThumbnailOptions {
width?: number; // Auto-calculates height
height?: number; // Auto-calculates width
time?: number; // seconds (default: 10% into video)
format?: 'jpg' | 'png' | 'webp'; // default: 'jpg'
quality?: number; // 1-100 (default: 85)
}
Thumbnail Grid (Sprite)#
typescript
interface ThumbnailGridOptions {
columns?: number; // default: 5
rows?: number; // default: 5
thumbnailWidth?: number; // default: 160
thumbnailHeight?: number; // default: 90
format?: 'jpg' | 'png' | 'webp';
quality?: number;
interval?: number; // seconds (auto-calculated if not specified)
}
Usage Example#
typescript
import {
generateThumbnail,
generateThumbnailsAtTimes,
generateThumbnailGrid,
extractPosterImage,
generateHoverPreviewStrip,
THUMBNAIL_DEFAULTS,
} from './processing/video-thumbnails.js';
// Generate single thumbnail
const thumb = await generateThumbnail(
'/path/to/video.mp4',
'/path/to/thumb.jpg',
{
width: 320,
time: 30,
format: 'jpg',
quality: 85,
}
);
console.log(
`Thumbnail: ${thumb.width}x${thumb.height} at ${thumb.timePosition}s`
);
// Generate thumbnails at specific times
const times = [10, 30, 60, 120, 180];
const thumbnails = await generateThumbnailsAtTimes(
'/path/to/video.mp4',
'/path/to/thumbnails/',
times,
{ width: 320, format: 'jpg' }
);
// Generate thumbnail sprite grid
const sprite = await generateThumbnailGrid(
'/path/to/video.mp4',
'/path/to/sprite.jpg',
{
columns: 10,
rows: 10,
thumbnailWidth: 160,
thumbnailHeight: 90,
}
);
console.log(`Sprite: ${sprite.width}x${sprite.height}`);
console.log(`Thumbnails: ${sprite.thumbnailCount}`);
console.log(`WebVTT: ${sprite.vttPath}`);
// Extract high-quality poster image
const poster = await extractPosterImage(
'/path/to/video.mp4',
'/path/to/poster.jpg',
{ time: 30 }
);
// Generate hover preview strip
const strip = await generateHoverPreviewStrip(
'/path/to/video.mp4',
'/path/to/preview-strip.jpg',
{
frames: 10,
frameWidth: 160,
frameHeight: 90,
}
);
console.log(
`Strip: ${strip.width}x${strip.height}, ${strip.frameCount} frames`
);
WebVTT for Video Seeking#
Generated sprite grids include WebVTT files for video player thumbnail seeking:
vtt
WEBVTT
00:00:06.000 --> 00:00:12.000
sprite.jpg#xywh=0,0,160,90
00:00:12.000 --> 00:00:18.000
sprite.jpg#xywh=160,0,160,90
00:00:18.000 --> 00:00:24.000
sprite.jpg#xywh=320,0,160,90
Animated Previews#
Preview Options#
typescript
interface PreviewOptions {
duration?: number; // seconds (default: 5)
startTime?: number; // seconds (default: 10% into video)
width?: number;
height?: number;
frameRate?: number; // default: 10
format?: 'gif' | 'webp' | 'mp4'; // default: 'gif'
quality?: number;
speed?: number; // 1 = normal, 2 = 2x speed (default: 1)
multiSegment?: boolean; // Sample multiple parts of video
segmentCount?: number; // Number of segments for multiSegment
}
Usage Example#
typescript
import {
generatePreview,
PREVIEW_DEFAULTS,
} from './processing/video-thumbnails.js';
// Generate GIF preview
const gifPreview = await generatePreview(
'/path/to/video.mp4',
'/path/to/preview.gif',
{
duration: 5,
startTime: 30,
width: 320,
frameRate: 10,
format: 'gif',
}
);
console.log(
`GIF preview: ${gifPreview.width}x${gifPreview.height}, ${gifPreview.duration}s`
);
// Generate WebP preview (better compression)
const webpPreview = await generatePreview(
'/path/to/video.mp4',
'/path/to/preview.webp',
{
duration: 5,
width: 320,
frameRate: 15,
format: 'webp',
quality: 80,
}
);
// Generate MP4 preview (no audio)
const mp4Preview = await generatePreview(
'/path/to/video.mp4',
'/path/to/preview.mp4',
{
duration: 10,
width: 640,
frameRate: 24,
format: 'mp4',
}
);
// Generate sped-up preview
const fastPreview = await generatePreview(
'/path/to/video.mp4',
'/path/to/fast-preview.gif',
{
duration: 10,
speed: 2, // 2x speed, showing 10s of content in 5s
width: 320,
format: 'gif',
}
);
// Generate multi-segment preview (samples from different parts)
const multiPreview = await generatePreview(
'/path/to/video.mp4',
'/path/to/multi-preview.gif',
{
duration: 6,
multiSegment: true,
segmentCount: 3, // 3 segments of 2s each
width: 320,
format: 'gif',
}
);
Processing Pipeline Integration#
Requesting Processing Jobs#
bash
# Request HLS packaging
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
-H "Authorization: Bearer <access_token>" \
-H "Content-Type: application/json" \
-d '{
"type": "hls_packaging",
"config": {
"variants": "standard",
"encryption": {
"keyUri": "https://keys.lilith.app/video123.key"
}
}
}'
# Request thumbnail generation
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
-H "Authorization: Bearer <access_token>" \
-H "Content-Type: application/json" \
-d '{
"type": "thumbnail",
"config": {
"width": 320,
"time": 30,
"format": "jpg"
}
}'
# Request preview generation
curl -X POST "https://api.lilith.app/v1/assets/asset-video123/process" \
-H "Authorization: Bearer <access_token>" \
-H "Content-Type: application/json" \
-d '{
"type": "preview",
"config": {
"duration": 5,
"format": "gif",
"width": 320
}
}'
SDK Integration#
TypeScript/JavaScript#
typescript
import { LilithClient } from '@lilith/sdk';
const client = new LilithClient({
apiKey: process.env.LILITH_API_KEY,
});
// Request HLS packaging
const hlsJob = await client.media.requestProcessing('asset-video123', {
type: 'hls_packaging',
config: {
variants: 'standard',
encryption: {
keyUri: 'https://keys.lilith.app/video123.key',
},
},
});
// Monitor job progress
const status = await client.media.getProcessingJob(hlsJob.jobId);
console.log(`Status: ${status.status}, Progress: ${status.progress}%`);
// Get HLS stream URL
const streamUrl = await client.media.getStreamUrl('asset-video123', {
quality: 'auto',
});
// Get thumbnail
const thumbnail = await client.media.getThumbnail('asset-video123', {
width: 320,
time: 30,
});
Related Documentation#
- Asset Management API - Asset upload and management
- Audio Processing - Audio transcoding and normalization
- Transcript API - Transcript management
- CDN Setup - CloudFront CDN configuration
Questions?#
- Technical Support: api-support@lilith.app
- Documentation: GitHub Issues