Aja is the motion capture and animation processing domain within the Oshun monorepo. It provides services and libraries for video-to-motion extraction, AI motion retargeting, skeletal animation, motion format conversion, and pipeline orchestration.
Architecture Overview#
apps/aja/
svc-motion-pipeline/ # Pipeline orchestration service (Fastify)
svc-motion-ai/ # AI motion processing service (Hono)
svc-reference-video/ # Reference video analysis service
cli/ # CLI tool for motion processing
libs/aja/
motion-formats/ # BVH, FBX, glTF, USD, Alembic, C3D, TRC, ASF/AMC
motion-processing/ # Core motion data transforms
motion-validation/ # Quality validation and checking
motion-quality/ # Quality metrics and scoring
motion-pipeline-sdk/ # TypeScript SDK for pipeline API
motion-pipeline-sdk-python/ # Python SDK for pipeline API
motion-integration/ # Integration adapters
skeleton-mapping/ # Skeleton topology mapping
pose-lifting/ # 2D-to-3D pose lifting
depth-sensing/ # Depth estimation
human-mesh-recovery/ # Full body mesh from video
neural-retargeting/ # Neural network motion retargeting
semantic-retargeting/ # Semantic-aware motion retargeting
blend-shape-retargeting/ # Facial blend shape retargeting
proportional-adaptation/ # Body proportion adaptation
optimization-ik/ # Inverse kinematics optimization
animation-blending/ # Animation clip blending
avatar-integration/ # Avatar system integration
avatar-library/ # Reusable avatar models
avatar-preview-ui/ # Avatar preview components
bone-mapping-ui/ # Bone mapping editor UI
domain-motion-pipelines/ # Domain-specific pipeline presets
fitness-animation/ # Fitness-specific motion processing
video-chunking/ # Video segmentation for processing
multi-view-reconstruction/ # Multi-camera reconstruction
batch-inference/ # Batch processing for ML models
pipeline-cache/ # Pipeline result caching
pipeline-parallelism/ # Parallel stage execution
result-aggregation/ # Multi-output aggregation
model-optimization/ # ML model optimization
asset-storage/ # Motion file storage
content-security/ # Content security scanning
content-moderation/ # Content moderation
content-watermarking/ # Watermarking for output files
consent-management/ # Consent tracking for captured performers
privacy-protection/ # PII detection/redaction in motion data
data-retention/ # Data lifecycle management
Prerequisites#
| Tool | Version | Purpose |
|---|---|---|
| Node.js | >= 18.x | Runtime |
| pnpm | >= 8.x | Package manager |
| FFmpeg | >= 5.x | Video decoding, frame extraction |
| Docker | >= 24.x | Optional: containerized GPU workers |
Install FFmpeg (Ubuntu/Debian):
sudo apt update && sudo apt install -y ffmpeg
Verify:
ffmpeg -version
node --version
pnpm --version
Install monorepo dependencies from the repo root:
pnpm install
Running Services Locally#
svc-motion-pipeline (Pipeline Orchestration)#
The pipeline service is a Fastify-based HTTP API that manages motion processing jobs: submission, configuration, status tracking, webhooks, SSE streaming, and download.
# Development mode (auto-reload via tsx watch)
pnpm nx dev @lilith/svc-motion-pipeline
# Production build and start
pnpm nx build @lilith/svc-motion-pipeline
pnpm nx start @lilith/svc-motion-pipeline
The server starts on the port defined by PORT (default: inferred by
@lilith/fastify-core). Key environment variables:
| Variable | Default | Description |
|---|---|---|
SERVICE_NAME |
motion-pipeline |
Service identifier |
PORT |
(fastify-core) | HTTP listen port |
NODE_ENV |
development |
Environment name |
API routes (all under /v1/pipeline/):
| Method | Path | Purpose |
|---|---|---|
| GET | /health |
Health check |
| GET | /v1/pipeline/stats |
Queue statistics |
| GET | /v1/pipeline/configs |
List pipeline configurations |
| GET | /v1/pipeline/configs/:id |
Get config by ID |
| GET | /v1/pipeline/configs/by-name/:n |
Get config by name |
| POST | /v1/pipeline/configs |
Create config (JSON) |
| POST | /v1/pipeline/configs/yaml |
Create config (YAML) |
| GET | /v1/pipeline/configs/:id/yaml |
Export config as YAML |
| DELETE | /v1/pipeline/configs/:id |
Delete config |
| GET | /v1/pipeline/presets |
List quality presets |
| POST | /v1/pipeline/jobs |
Submit a job |
| GET | /v1/pipeline/jobs |
List jobs |
| GET | /v1/pipeline/jobs/:id |
Get job (polling) |
| POST | /v1/pipeline/jobs/:id/cancel |
Cancel a job |
| POST | /v1/pipeline/jobs/:id/retry |
Retry a failed job |
| PATCH | /v1/pipeline/jobs/:id/priority |
Update job priority |
| POST | /v1/pipeline/jobs/batch |
Batch operations |
| GET | /v1/pipeline/jobs/:id/outputs |
List job outputs |
| POST | /v1/pipeline/jobs/:id/download |
Generate download URL |
| GET | /v1/pipeline/jobs/:id/webhooks |
Webhook deliveries for job |
| GET | /v1/pipeline/jobs/:id/stream |
SSE progress stream |
svc-motion-ai (AI Processing)#
The motion AI service is a Hono-based HTTP API that runs the actual AI motion processing: video-to-motion, retargeting, skeletal animation, and video analysis.
# Development mode (auto-rebuild via nx serve)
pnpm nx serve aja-svc-motion-ai
# Run tests
pnpm nx test aja-svc-motion-ai
Default port: 3040 (configurable via PORT environment variable).
API routes:
| Method | Path | Purpose |
|---|---|---|
| GET | /health |
Health check |
| POST | /api/v1/video-to-motion/process |
Extract motion from video |
| POST | /api/v1/retargeting/retarget |
Retarget motion data |
| POST | /api/v1/skeletal/animate |
Skeletal animation |
| POST | /api/v1/analysis/analyze |
Video analysis |
Modules inside svc-motion-ai:
- video-to-motion/ - Video preprocessing, pose estimation, temporal
processing, keypoint normalization, and the main
VideoToMotionService. - motion-retargeting/ - Motion retargeting between different skeleton
topologies via
MotionRetargetingService. - skeletal-animation-motion/ - Skeletal animation processing via
SkeletalAnimationMotionService. - video-analysis/ - Video analysis via
VideoAnalysisService. - motion-enhancement/ - Super-resolution, style transfer, synthesis, and physics refinement.
- live-motion-capture/ - Real-time pose estimation, webcam capture, live avatar driving, and streaming pose lift.
- mobile-live-capture/ - iOS/Android on-device capture and streaming.
CLI#
The Aja CLI provides command-line access to motion processing capabilities:
pnpm nx build aja-cli
# Available commands:
# process - Process a video through the pipeline
# convert - Convert between motion formats
# inspect - Inspect a motion file
# jobs - Manage pipeline jobs
# health - Check service health
# config - Manage pipeline configurations
# debug - Debug utilities
Pipeline Configuration and Quality Presets#
Pipeline configurations define which stages run, their parameters, quality
settings, input constraints, and output specifications. They can be created via
the JSON API, YAML endpoint, or the PipelineBuilder programmatic API.
Quality Presets#
Four built-in quality presets control the accuracy/speed trade-off:
| Preset | Description |
|---|---|
draft |
Fast processing, lower quality. Good for previews. |
standard |
Balanced quality and speed. Default for most jobs. |
high |
Higher quality with more processing time. |
ultra |
Maximum quality. Full pipeline with all refinements. |
Pipeline Domains#
Pipeline configurations are scoped to motion domains:
yoga- Yoga pose tracking and analysisfitness- General fitness motion capturedance- Dance choreography capturemartial-arts- Martial arts form capturegeneral- General-purpose motion capture
On startup, the service auto-generates default configurations for every domain/preset combination (20 total: 5 domains x 4 presets).
YAML Pipeline Definition#
Pipelines can be defined in YAML and uploaded via
POST /v1/pipeline/configs/yaml:
name: custom-yoga-pipeline
description: Custom yoga pipeline with extra smoothing
domain: yoga
quality: high
stages:
- name: pose-estimation
enabled: true
timeoutMs: 30000
- name: motion-refinement
enabled: true
timeoutMs: 20000
- name: ik-optimization
enabled: true
timeoutMs: 15000
Programmatic Configuration#
import { createPipelineBuilder } from '@lilith/svc-motion-pipeline';
const config = createPipelineBuilder()
.name('my-custom-pipeline')
.description('Custom pipeline for dance motion')
.domain('dance')
.quality('high')
.build();
Testing#
Unit Tests#
# Run all Aja motion-ai tests
pnpm nx test aja-svc-motion-ai
# Run all svc-motion-pipeline tests
pnpm nx test @lilith/svc-motion-pipeline
# Run tests for a specific library
pnpm nx test @aja/motion-formats
pnpm nx test @aja/motion-processing
# Run tests with coverage
pnpm nx test aja-svc-motion-ai -- --coverage
# Run a specific test file
pnpm nx test aja-svc-motion-ai -- --testPathPattern="video-analysis"
Vitest Configuration#
Tests use Vitest with the following configuration (from
apps/aja/svc-motion-ai/vitest.config.ts):
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
globals: true,
environment: 'node',
include: ['src/**/*.{test,spec}.ts'],
coverage: {
provider: 'v8',
reporter: ['text', 'json', 'html'],
},
},
});
Test Fixtures#
Test files live alongside source code in __tests__/ directories:
apps/aja/svc-motion-ai/src/__tests__/
video-analysis/service.test.ts
motion-retargeting/motion-retargeting-service.test.ts
skeletal-animation-motion/skeletal-animation-motion-service.test.ts
Linting#
pnpm nx lint aja-svc-motion-ai
pnpm nx lint @lilith/svc-motion-pipeline
Type Checking#
pnpm nx typecheck @lilith/svc-motion-pipeline
How to Add a New Motion Format#
The @aja/motion-formats library is the central format conversion layer. It
supports BVH, FBX, glTF/GLB, USD/USDA/USDC/USDZ, Alembic, C3D, TRC, and ASF/AMC.
To add a new format:
1. Create the Format Module#
Create a new file in libs/aja/motion-formats/src/:
// libs/aja/motion-formats/src/my-format.ts
import type { AnimationClip, ConversionResult } from './types.js';
export interface MyFormatImportOptions {
// Format-specific options
scale?: number;
}
export interface MyFormatExportOptions {
// Format-specific options
version?: string;
}
export class MyFormatImporter {
constructor(private options?: MyFormatImportOptions) {}
import(data: string | Uint8Array): ConversionResult<AnimationClip> {
// Parse the format into an AnimationClip
// Return { success: true, data: clip } or { success: false, errors: [...] }
}
}
export class MyFormatExporter {
constructor(private options?: MyFormatExportOptions) {}
export(clip: AnimationClip): ConversionResult<string | Uint8Array> {
// Convert AnimationClip to the target format
// Return { success: true, data: outputData } or { success: false, errors: [...] }
}
}
2. Register in the Index#
Add exports in libs/aja/motion-formats/src/index.ts:
export { MyFormatImporter, MyFormatExporter } from './my-format.js';
3. Add to Convenience Functions#
Update importAnimation(), exportAnimation(), and detectFormat() in
libs/aja/motion-formats/src/index.ts to handle the new format string.
4. Add to the MotionFormat Type#
In libs/aja/motion-formats/src/types.ts, add the new format to the
MotionFormat union type:
export type MotionFormat = 'bvh' | 'fbx' | 'gltf' | ... | 'my-format';
5. Write Tests#
Create libs/aja/motion-formats/src/__tests__/my-format.test.ts covering
import, export, round-trip conversion, edge cases, and error handling.
How to Add a New Pipeline Stage#
Pipeline stages are defined in the svc-motion-pipeline configuration system.
1. Define the Stage#
Add the stage name to the PipelineStage type in
apps/aja/svc-motion-pipeline/src/types.ts:
export type PipelineStage =
| 'pose-estimation'
| 'motion-refinement'
| ...
| 'my-new-stage';
2. Add Default Configuration#
In the pipeline configuration module
(apps/aja/svc-motion-pipeline/src/config/), add the default timeout, retry
settings, and quality-preset-specific parameters for the new stage.
3. Add Stage Dependencies#
If the stage depends on other stages completing first, add the dependency in
STAGE_DEPENDENCIES. If the stage can run in parallel with others, add it to
STAGE_PARALLELISM.
4. Implement the Stage Processor#
The job processor in app.ts iterates through enabled stages. To add real
processing logic for the new stage, implement it either:
- As a function in
svc-motion-pipelinethat the processor calls for your stage name. - As an external call to
svc-motion-aior another service endpoint.
5. Add to Quality Presets#
Each quality preset (DRAFT_PRESET, STANDARD_PRESET, HIGH_PRESET,
ULTRA_PRESET) should specify whether the new stage is enabled and its
quality-specific parameters.
6. Update Tests#
Add test cases for the new stage in pipeline configuration validation, job processing, and integration tests.
Key Libraries Reference#
| Library | Package | Purpose |
|---|---|---|
motion-formats |
@aja/motion-formats |
Format I/O (BVH, FBX, glTF...) |
motion-processing |
@aja/motion-processing |
Core transforms and filters |
motion-validation |
@aja/motion-validation |
Quality validation |
motion-quality |
@aja/motion-quality |
Quality metrics and scoring |
skeleton-mapping |
@aja/skeleton-mapping |
Skeleton topology mapping |
pose-lifting |
@aja/pose-lifting |
2D-to-3D pose lifting |
optimization-ik |
@aja/optimization-ik |
IK solvers |
animation-blending |
@aja/animation-blending |
Clip blending |
pipeline-cache |
@aja/pipeline-cache |
Pipeline result caching |
pipeline-parallelism |
@aja/pipeline-parallelism |
Parallel stage execution |
motion-pipeline-sdk |
@aja/motion-pipeline-sdk |
TS SDK for pipeline API |
motion-pipeline-sdk-python |
aja_motion_pipeline |
Python SDK for pipeline API |
Troubleshooting#
FFmpeg not found#
The video processing stages require FFmpeg on the system PATH. Verify with:
which ffmpeg
ffmpeg -version
Port already in use#
If svc-motion-ai fails to start on port 3040:
lsof -i :3040
kill -9 <PID>
Out of memory during video processing#
Large video files can exhaust memory. Use the video-chunking library to
segment videos before processing, or adjust Node.js memory limits:
NODE_OPTIONS="--max-old-space-size=8192" pnpm nx serve aja-svc-motion-ai