Canonical source: the canonical architecture document for this domain is
deep-dive/architecture.md , rendered in the
docs center. This file is its operational companion (service topology, APIs,
schemas, runtime detail) — reconciled by linking, not duplicated (§2, §13).
Aja -- Motion AI and Animation Pipeline Platform
Named after the Yoruba Orisha Aja (forest guide), this domain provides an
end-to-end motion processing pipeline: from raw video or motion capture input
through pose estimation, skeletal retargeting, quality assessment, and format
conversion to production-ready animation files.
System Overview#
Aja is a motion-intelligence platform consisting of 4 service applications
and 38 libraries organized into 7 functional categories. The platform
accepts video, motion capture data, or live camera feeds and transforms them
into clean, retargeted animation data in industry-standard formats.
Core Value Proposition#
Video-to-Motion -- Extract skeletal animation from ordinary video using
AI pose estimation, depth sensing, and human mesh recovery.
Motion Quality -- Automatically assess, clean, and validate motion data
through noise reduction, physics cleanup, foot sliding correction, and QA
pipelines.
Format Universality -- Import and export across 13+ animation formats
(BVH, FBX, GLB, USD, Alembic, C3D, TRC, ASF/AMC, Collada, MDD, PC2, OBJ
sequences, custom JSON).
Domain Specialization -- Pre-configured pipelines for yoga, fitness,
dance, martial arts, and general motion with domain-specific analysis.
Service Architecture#
Applications#
text Copy
apps/aja/
svc-motion-pipeline/ Fastify :8090 Core pipeline orchestration
svc-motion-ai/ Hono :3040 AI motion processing
svc-reference-video/ (library) -- Video reference management
cli/ Commander -- Command-line tools
Application
Framework
Port
Description
svc-motion-pipeline
Fastify
8090
Pipeline orchestration, job management, configuration, result delivery (webhooks, SSE, polling, downloads)
svc-motion-ai
Hono
3040
AI-powered motion processing: video-to-motion conversion, retargeting, skeletal animation, live capture, mobile capture, motion enhancement, video analysis
svc-reference-video
-- (library)
--
Reference video library: ingestion, metadata extraction, scene detection, categorization, search (CLIP/vector), annotation, collaboration
cli
Commander.js
--
CLI tool (aja / aja-motion) with commands: process, convert, inspect, debug, config, jobs, health
svc-motion-pipeline -- API Endpoints#
This is the primary orchestration service. Built on Fastify with
@lilith/fastify-core.
Health and Status:
Method
Path
Description
GET
/health
Service health (components, uptime)
GET
/v1/pipeline/stats
Queue statistics (waiting, active, completed, failed)
Pipeline Configuration:
Method
Path
Description
GET
/v1/pipeline/configs
List configurations (filter by domain, quality preset)
GET
/v1/pipeline/configs/:configId
Get configuration by ID
GET
/v1/pipeline/configs/by-name/:name
Get configuration by name
POST
/v1/pipeline/configs
Create configuration (JSON)
POST
/v1/pipeline/configs/yaml
Create configuration (YAML)
GET
/v1/pipeline/configs/:configId/yaml
Export configuration as YAML
DELETE
/v1/pipeline/configs/:configId
Delete configuration
GET
/v1/pipeline/presets
List quality presets (draft, standard, high, ultra)
Job Management:
Method
Path
Description
POST
/v1/pipeline/jobs
Submit processing job (returns 202 with pollUrl/streamUrl)
GET
/v1/pipeline/jobs
List jobs (filter by status, domain, tags; pagination)
GET
/v1/pipeline/jobs/:jobId
Get job status (polling endpoint)
POST
/v1/pipeline/jobs/:jobId/cancel
Cancel job
POST
/v1/pipeline/jobs/:jobId/retry
Retry failed job
PATCH
/v1/pipeline/jobs/:jobId/priority
Update job priority
POST
/v1/pipeline/jobs/batch
Batch operations (cancel, retry, prioritize)
Result Delivery:
Method
Path
Description
GET
/v1/pipeline/jobs/:jobId/outputs
List available outputs (completed jobs)
POST
/v1/pipeline/jobs/:jobId/download
Generate signed download URL
GET
/v1/pipeline/jobs/:jobId/stream
SSE progress stream
GET
/v1/pipeline/streams/stats
Active stream connection count
Webhooks:
Method
Path
Description
GET
/v1/pipeline/jobs/:jobId/webhooks
Get webhook deliveries for job
GET
/v1/pipeline/webhooks/:deliveryId
Get webhook delivery status
POST
/v1/pipeline/webhooks/:deliveryId/retry
Retry webhook delivery
GET
/v1/pipeline/webhooks/stats
Webhook service statistics
svc-motion-ai -- API Endpoints#
Built on Hono with CORS and logger middleware.
Method
Path
Description
GET
/health
Health check
POST
/api/v1/video-to-motion/process
Convert video to motion data
POST
/api/v1/retargeting/retarget
Retarget motion to different skeleton
POST
/api/v1/skeletal/animate
Generate skeletal animation
POST
/api/v1/analysis/analyze
Analyze video content
The service internally contains 7 modules:
video-to-motion -- Video preprocessing, pose estimation engine, keypoint
normalization, temporal processing
live-motion-capture -- Webcam capture, realtime pose estimation, realtime
retargeting, streaming pose lift, live avatar driving
mobile-live-capture -- iOS capture, Android capture, on-device processing,
mobile streaming
motion-retargeting -- Skeleton-to-skeleton retargeting service
motion-enhancement -- Motion super-resolution, style transfer, physics
refinement, motion synthesis
skeletal-animation-motion -- Skeletal animation generation
video-analysis -- Video content analysis
CLI Commands#
The aja CLI connects to the pipeline service via @aja/motion-pipeline-sdk.
Command
Description
aja process [input...]
Submit videos for motion extraction (supports glob, URL, interactive mode)
aja convert <input> <output>
Batch format conversion between motion formats
aja inspect <file>
Quality inspection and validation of motion files
aja debug
Pipeline debugging and troubleshooting
aja config
Manage pipeline configurations
aja jobs
View and manage processing jobs
aja health
Check API service health
Library Architecture#
Dependency Graph (Simplified)#
text Copy
+--------------------+
| svc-motion-pipeline|
+---------+----------+
|
+------------+------------+------------+
| | | |
motion-formats motion- motion- domain-motion-
processing quality pipelines
| | | |
+------+-----+ | +------+------+
| | | | |
motion-validation | yoga fitness dance martial-arts
|
+---------+---------+
| | |
skeleton- optimization- avatar-
mapping ik library
+------------------+
| svc-motion-ai |
+---------+--------+
|
+----------+----------+-------+--------+----------+
| | | | | |
pose- depth- human-mesh multi- neural- motion-formats
lifting sensing recovery view- retargeting
recon
Category 1: Core Motion Processing (5 libs)#
Library
Package
Purpose
motion-formats
@aja/motion-formats
Multi-format import/export with parsers/writers for BVH, FBX, GLB/glTF, USD/USDA/USDC/USDZ, Alembic, C3D, TRC, ASF/AMC. Includes importAnimation(), exportAnimation(), convertAnimation(), detectFormat(), and motion clip database.
motion-processing
@aja/motion-processing
6-stage processing pipeline: noise reduction (FFT, wavelets, adaptive), foot sliding correction (IK solvers: TwoBone, FABRIK, CCD), physics cleanup (CoM, angular momentum, balance, joint torque limits), retiming (time warps, beat sync), gap filling (short/long, keyframe insertion), segmentation (action, transitions, loop points). Exposed as MotionProcessingPipeline class and quickCleanup()/fullProcessing() convenience functions.
motion-quality
@aja/motion-quality
Quality assessment: basic metrics (jitter, foot sliding, bone length, joint angles, smoothness, plausibility), perceptual metrics (naturalness, style consistency, temporal coherence, visual quality), ground truth comparison (MPJPE, PA-MPJPE, PCK, AUC), and automated QA pipeline with pass/reject/review decisions.
motion-validation
@aja/motion-validation
Data validation and error detection for motion data.
motion-integration
@aja/motion-integration
Integration layer connecting processing stages.
Category 2: Video Analysis (4 libs)#
Library
Package
Purpose
pose-lifting
@aja/pose-lifting
2D-to-3D pose lifting from single or multi-view video
depth-sensing
@aja/depth-sensing
Monocular and stereo depth estimation
multi-view-reconstruction
@aja/multi-view-reconstruction
Multi-camera 3D reconstruction
human-mesh-recovery
@aja/human-mesh-recovery
SMPL/SMPL-X parametric mesh recovery from video
Category 3: Skeletal and Avatar (6 libs)#
Library
Package
Purpose
skeleton-mapping
@aja/skeleton-mapping
Skeleton hierarchy conversion between different standards
optimization-ik
@aja/optimization-ik
Inverse kinematics solvers (analytical + iterative)
avatar-library
@aja/avatar-library
Avatar definitions and presets
avatar-integration
@aja/avatar-integration
Avatar binding and animation
avatar-preview-ui
@aja/avatar-preview-ui
React components for avatar preview
bone-mapping-ui
@aja/bone-mapping-ui
Interactive bone mapping UI
Category 4: Retargeting and Blending (5 libs)#
Library
Package
Purpose
semantic-retargeting
@aja/semantic-retargeting
Semantic-aware motion retargeting
proportional-adaptation
@aja/proportional-adaptation
Body proportion adaptation across different morphologies
blend-shape-retargeting
@aja/blend-shape-retargeting
Facial blend shape retargeting
neural-retargeting
@aja/neural-retargeting
ML-based retargeting using neural networks
animation-blending
@aja/animation-blending
Motion blending, transitions, and interpolation
Category 5: Domain Pipelines (2 libs)#
Library
Package
Purpose
domain-motion-pipelines
@aja/domain-motion-pipelines
Factory for domain-specific pipelines (yoga, fitness, dance, martial-arts). Each pipeline provides detection, recognition, form analysis, and tracking specific to its domain.
fitness-animation
@aja/fitness-animation
Fitness-specific analysis (exercise recognition, rep counting, form analysis)
Domain Pipeline Capabilities:
Pipeline
Key Services
Yoga
Asana detection/recognition, alignment analysis, breath detection, drishti tracking, bandha analysis, meditation state detection, pranayama recognition, sequence tracking
Fitness
Exercise recognition, rep counting with phase detection, form analysis with real-time feedback, workout tracking (sets/reps), equipment detection
Dance
Style detection (ballet, hip-hop, salsa, etc.), beat detection, beat-movement synchronization, choreography segmentation, expression analysis, musicality scoring
Martial Arts
Style detection (boxing, karate, BJJ, etc.), strike detection/classification, power/speed estimation, stance analysis, guard position analysis, footwork patterns, combat sequence tracking
Category 6: Infrastructure (8 libs)#
Library
Package
Purpose
motion-pipeline-sdk
@aja/motion-pipeline-sdk
TypeScript SDK for pipeline service API
motion-pipeline-sdk-python
@aja/motion-pipeline-sdk-python
Python SDK with Jupyter notebook support
batch-inference
@aja/batch-inference
Batch processing for large motion datasets
video-chunking
@aja/video-chunking
Video splitting for parallel processing
model-optimization
@aja/model-optimization
ML model compression and optimization
pipeline-parallelism
@aja/pipeline-parallelism
Distributed pipeline stage execution
distributed-workers
@aja/distributed-workers
Worker coordination for distributed processing
result-aggregation
@aja/result-aggregation
Result combining from distributed workers
Category 7: Security and Compliance (6 libs)#
Library
Package
Purpose
privacy-protection
@aja/privacy-protection
Privacy-preserving processing (face blurring, body anonymization)
consent-management
@aja/consent-management
Consent tracking and enforcement
content-security
@aja/content-security
Security protocols for motion data
content-watermarking
@aja/content-watermarking
Digital watermarking for motion files
content-moderation
@aja/content-moderation
Content filtering and safety
data-retention
@aja/data-retention
Retention policy enforcement
Utility (1 lib)#
Library
Package
Purpose
pipeline-cache
@aja/pipeline-cache
LRU caching for pipeline stages
Data Flow#
Video-to-Animation Pipeline#
text Copy
Input Video Analysis Motion Processing
====== ============== ================
+--------+ chunk +-------------+ raw +-------------+
| Video | --------->| Pose |-------->| Noise |
| File | split | Estimation | poses | Reduction |
+--------+ +-------------+ +------+------+
| |
+--------+ +------v------+ +------v------+
| MoCap | | Depth | | Gap |
| C3D/TRC| --------->| Sensing | | Filling |
+--------+ import +------+------+ +------+------+
| |
+--------+ +------v------+ +------v------+
| Anim | | Human Mesh | | Foot Slide |
| BVH/FBX| --------->| Recovery | | Correction |
+--------+ import +------+------+ +------+------+
| |
+--------+ +------v------+ +------v------+
| Live | stream | Multi-View | | Physics |
| Camera | --------->| Reconstruct | | Cleanup |
+--------+ +-------------+ +------+------+
|
Retargeting Domain Analysis Output
=========== =============== ======
+-------------+ +-------------+ +--------+
| Skeleton |-------->| Yoga/Fitness/|------->| BVH |
| Mapping | | Dance/ | +--------+
+------+------+ | Martial Arts | +--------+
| +------+------+ export | FBX |
+------v------+ | ----->| +--------+
| Proportional| +------v------+ +--------+
| Adaptation | | Quality | | GLB |
+------+------+ | Assessment | +--------+
| +------+------+ +--------+
+------v------+ | | USD |
| Neural | +------v------+ +--------+
| Retarget | | QA Pipeline | +--------+
+------+------+ | Pass/Reject | |Alembic |
| +-------------+ +--------+
+------v------+
| Animation |
| Blending |
+-------------+
Job Lifecycle#
text Copy
Client svc-motion-pipeline svc-motion-ai
====== =================== =============
POST /jobs --------> JobManager.submit()
|
v
PriorityQueue.enqueue()
|
v (processing loop)
JobProcessor.processJob()
|
+-- updateStage("ingestion")
+-- updateStage("validation")
+-- updateStage("pose-estimation") --> AI Service
+-- updateStage("domain-analysis")
+-- updateStage("quality-assessment")
+-- updateStage("retargeting")
+-- updateStage("format-conversion")
|
v
completeJob(result)
|
+---> Webhook delivery
+---> SSE stream
+---> Poll response
+---> Signed download URL
Pipeline Stages#
The processing pipeline consists of up to 11 configurable stages:
Stage
Description
Configurable
ingestion
Accept and validate input files
timeout, retry
validation
Validate data integrity and format
timeout, retry
preprocessing
Prepare data for processing
timeout, retry
pose-estimation
AI pose estimation from video
timeout, retry, parallel
skeleton-fitting
Fit skeleton to estimated poses
timeout, retry
domain-analysis
Domain-specific analysis (yoga/fitness/dance/martial-arts)
timeout, retry
quality-assessment
Quality metrics and QA decisions
timeout, retry
retargeting
Map motion to target skeleton
timeout, retry
format-conversion
Convert to output formats
timeout, retry
postprocessing
Final cleanup and optimization
timeout, retry
delivery
Store results and notify
timeout, retry
Each stage supports:
Enable/disable toggle
Per-stage timeout
Retry with exponential/linear/fixed backoff
Stage dependencies (must-complete-before)
Parallel execution flag
Quality Presets#
Preset
Processing Time
Target Quality
Frame Rate
Key Features
draft
0.5x baseline
0.6
24 fps
Basic noise reduction
standard
1.0x baseline
0.8
30 fps
Noise reduction + smoothing + jitter correction
high
2.0x baseline
0.9
60 fps
All standard + foot sliding + contact detection + physics
ultra
4.0x baseline
0.95
120 fps
All high + IK cleanup + multi-view fusion
Cross-Domain Dependencies#
Shared @oshun/* Libraries Used#
Library
Used By
@oshun/cache
svc-motion-pipeline
@oshun/database
svc-motion-pipeline
@oshun/event-bus
svc-motion-pipeline
@oshun/queue
svc-motion-pipeline
@oshun/config
svc-motion-ai
@oshun/logging
svc-motion-ai
@oshun/errors
svc-motion-ai
@oshun/health
svc-motion-ai
Shared @lilith/* Libraries Used#
Library
Used By
@lilith/fastify-core
svc-motion-pipeline (service server bootstrap)
Technology Stack#
Layer
Technologies
Pipeline Service
Fastify 4.x, @lilith/fastify-core, uuid, js-yaml, zod
AI Service
Hono, @hono/node-server
CLI
Commander.js 12.x, chalk 5.x, inquirer 9.x, ora 8.x, yaml
SDKs
TypeScript SDK, Python SDK with Jupyter support
Motion Formats
Custom parsers/writers for 13+ formats
ML Models
Pose estimation, depth sensing, mesh recovery (SMPL-X)
Job Management
In-memory priority queue with event-driven processing
Result Delivery
Webhooks (HMAC signed), SSE streaming, polling, signed download URLs
Build
Nx with @nx/js:tsc and nx:run-commands
Testing
Vitest
Runtime
Node.js with ESM modules
Deployment Architecture#
text Copy
+------------------+
| Load Balancer |
+--------+---------+
|
+--------------+--------------+
| |
+--------v--------+ +--------v--------+
| svc-motion- | | svc-motion-ai |
| pipeline :8090 | | :3040 |
+--------+--------+ +--------+--------+
| |
| +-----------+ |
+-->| Redis | |
| | (queue/ | |
| | cache) | |
| +-----------+ |
| |
| +-----------+ |
+-->| S3/MinIO |<-----------+
| | (storage) |
| +-----------+
|
| +-----------+
+-->| PostgreSQL|
| (jobs/ |
| configs) |
+-----------+
Services are deployed via the shared ECS deployment pipeline (deploy-ecs.yml).
The pipeline service coordinates job execution and stores results in
S3-compatible storage. Redis handles job queuing and caching. PostgreSQL stores
job metadata and pipeline configurations.
Configuration#
Environment Variables#
bash Copy
# svc-motion-pipeline
PORT=8090 # Service port
HOST=0.0.0.0 # Bind address
MAX_CONCURRENT_JOBS=10 # Concurrent job limit
DEFAULT_JOB_TIMEOUT_MS=1800000 # 30-minute default timeout
# svc-motion-ai
PORT=3040 # Service port
# CLI
AJA_API_URL=http://localhost:3000 # Pipeline API URL
AJA_API_KEY=... # API key for authentication
Pipeline Configuration (YAML)#
Pipelines can be defined as YAML:
yaml Copy
apiVersion: v1
kind: PipelineDefinition
metadata:
name: yoga-high-quality
labels:
domain: yoga
quality: high
spec:
domain: yoga
qualityPreset: high
stages:
- name: ingestion
type: ingestion
enabled: true
timeout: 60s
retry:
maxAttempts: 3
backoff: exponential
initialDelay: 1s
maxDelay: 30s
- name: pose-estimation
type: pose-estimation
enabled: true
dependsOn: [ingestion]
timeout: 300s
input:
formats: [mp4, mov, avi]
frameRate:
min: 24
max: 120
duration:
min: 1s
max: 3600s
maxSize: 2GB
output:
format: fbx
additionalFormats: [bvh, glb]
compression: gzip
includeAnalytics: true
settings:
timeout: 1800s
failFast: true
Development#
bash Copy
# Start pipeline service in dev mode
pnpm nx dev @aja/svc-motion-pipeline
# Start AI service in dev mode
pnpm nx dev aja-svc-motion-ai
# Run CLI
pnpm nx dev @aja/cli -- process video.mp4 --preset high --domain yoga
# Run tests for specific lib
pnpm nx test @aja/motion-formats
# Build all Aja projects
pnpm nx run-many --target=build --projects=tag:scope:aja
# Lint all Aja projects
pnpm nx run-many --target=lint --projects=tag:scope:aja