Eve library

Aja Domain - Developer Guide

Guide

Revision 1

On this page

Aja is the motion capture and animation processing domain within the Oshun monorepo. It provides services and libraries for video-to-motion extraction, AI motion retargeting, skeletal animation, motion format conversion, and pipeline orchestration.

Architecture Overview#

text
apps/aja/
  svc-motion-pipeline/   # Pipeline orchestration service (Fastify)
  svc-motion-ai/         # AI motion processing service (Hono)
  svc-reference-video/   # Reference video analysis service
  cli/                   # CLI tool for motion processing

libs/aja/
  motion-formats/        # BVH, FBX, glTF, USD, Alembic, C3D, TRC, ASF/AMC
  motion-processing/     # Core motion data transforms
  motion-validation/     # Quality validation and checking
  motion-quality/        # Quality metrics and scoring
  motion-pipeline-sdk/   # TypeScript SDK for pipeline API
  motion-pipeline-sdk-python/  # Python SDK for pipeline API
  motion-integration/    # Integration adapters
  skeleton-mapping/      # Skeleton topology mapping
  pose-lifting/          # 2D-to-3D pose lifting
  depth-sensing/         # Depth estimation
  human-mesh-recovery/   # Full body mesh from video
  neural-retargeting/    # Neural network motion retargeting
  semantic-retargeting/  # Semantic-aware motion retargeting
  blend-shape-retargeting/  # Facial blend shape retargeting
  proportional-adaptation/  # Body proportion adaptation
  optimization-ik/       # Inverse kinematics optimization
  animation-blending/    # Animation clip blending
  avatar-integration/    # Avatar system integration
  avatar-library/        # Reusable avatar models
  avatar-preview-ui/     # Avatar preview components
  bone-mapping-ui/       # Bone mapping editor UI
  domain-motion-pipelines/  # Domain-specific pipeline presets
  fitness-animation/     # Fitness-specific motion processing
  video-chunking/        # Video segmentation for processing
  multi-view-reconstruction/  # Multi-camera reconstruction
  batch-inference/       # Batch processing for ML models
  pipeline-cache/        # Pipeline result caching
  pipeline-parallelism/  # Parallel stage execution
  result-aggregation/    # Multi-output aggregation
  model-optimization/    # ML model optimization
  asset-storage/         # Motion file storage
  content-security/      # Content security scanning
  content-moderation/    # Content moderation
  content-watermarking/  # Watermarking for output files
  consent-management/    # Consent tracking for captured performers
  privacy-protection/    # PII detection/redaction in motion data
  data-retention/        # Data lifecycle management

Prerequisites#

Tool Version Purpose
Node.js >= 18.x Runtime
pnpm >= 8.x Package manager
FFmpeg >= 5.x Video decoding, frame extraction
Docker >= 24.x Optional: containerized GPU workers

Install FFmpeg (Ubuntu/Debian):

bash
sudo apt update && sudo apt install -y ffmpeg

Verify:

bash
ffmpeg -version
node --version
pnpm --version

Install monorepo dependencies from the repo root:

bash
pnpm install

Running Services Locally#

svc-motion-pipeline (Pipeline Orchestration)#

The pipeline service is a Fastify-based HTTP API that manages motion processing jobs: submission, configuration, status tracking, webhooks, SSE streaming, and download.

bash
# Development mode (auto-reload via tsx watch)
pnpm nx dev @lilith/svc-motion-pipeline

# Production build and start
pnpm nx build @lilith/svc-motion-pipeline
pnpm nx start @lilith/svc-motion-pipeline

The server starts on the port defined by PORT (default: inferred by @lilith/fastify-core). Key environment variables:

Variable Default Description
SERVICE_NAME motion-pipeline Service identifier
PORT (fastify-core) HTTP listen port
NODE_ENV development Environment name

API routes (all under /v1/pipeline/):

Method Path Purpose
GET /health Health check
GET /v1/pipeline/stats Queue statistics
GET /v1/pipeline/configs List pipeline configurations
GET /v1/pipeline/configs/:id Get config by ID
GET /v1/pipeline/configs/by-name/:n Get config by name
POST /v1/pipeline/configs Create config (JSON)
POST /v1/pipeline/configs/yaml Create config (YAML)
GET /v1/pipeline/configs/:id/yaml Export config as YAML
DELETE /v1/pipeline/configs/:id Delete config
GET /v1/pipeline/presets List quality presets
POST /v1/pipeline/jobs Submit a job
GET /v1/pipeline/jobs List jobs
GET /v1/pipeline/jobs/:id Get job (polling)
POST /v1/pipeline/jobs/:id/cancel Cancel a job
POST /v1/pipeline/jobs/:id/retry Retry a failed job
PATCH /v1/pipeline/jobs/:id/priority Update job priority
POST /v1/pipeline/jobs/batch Batch operations
GET /v1/pipeline/jobs/:id/outputs List job outputs
POST /v1/pipeline/jobs/:id/download Generate download URL
GET /v1/pipeline/jobs/:id/webhooks Webhook deliveries for job
GET /v1/pipeline/jobs/:id/stream SSE progress stream

svc-motion-ai (AI Processing)#

The motion AI service is a Hono-based HTTP API that runs the actual AI motion processing: video-to-motion, retargeting, skeletal animation, and video analysis.

bash
# Development mode (auto-rebuild via nx serve)
pnpm nx serve aja-svc-motion-ai

# Run tests
pnpm nx test aja-svc-motion-ai

Default port: 3040 (configurable via PORT environment variable).

API routes:

Method Path Purpose
GET /health Health check
POST /api/v1/video-to-motion/process Extract motion from video
POST /api/v1/retargeting/retarget Retarget motion data
POST /api/v1/skeletal/animate Skeletal animation
POST /api/v1/analysis/analyze Video analysis

Modules inside svc-motion-ai:

  • video-to-motion/ - Video preprocessing, pose estimation, temporal processing, keypoint normalization, and the main VideoToMotionService.
  • motion-retargeting/ - Motion retargeting between different skeleton topologies via MotionRetargetingService.
  • skeletal-animation-motion/ - Skeletal animation processing via SkeletalAnimationMotionService.
  • video-analysis/ - Video analysis via VideoAnalysisService.
  • motion-enhancement/ - Super-resolution, style transfer, synthesis, and physics refinement.
  • live-motion-capture/ - Real-time pose estimation, webcam capture, live avatar driving, and streaming pose lift.
  • mobile-live-capture/ - iOS/Android on-device capture and streaming.

CLI#

The Aja CLI provides command-line access to motion processing capabilities:

bash
pnpm nx build aja-cli

# Available commands:
#   process   - Process a video through the pipeline
#   convert   - Convert between motion formats
#   inspect   - Inspect a motion file
#   jobs      - Manage pipeline jobs
#   health    - Check service health
#   config    - Manage pipeline configurations
#   debug     - Debug utilities

Pipeline Configuration and Quality Presets#

Pipeline configurations define which stages run, their parameters, quality settings, input constraints, and output specifications. They can be created via the JSON API, YAML endpoint, or the PipelineBuilder programmatic API.

Quality Presets#

Four built-in quality presets control the accuracy/speed trade-off:

Preset Description
draft Fast processing, lower quality. Good for previews.
standard Balanced quality and speed. Default for most jobs.
high Higher quality with more processing time.
ultra Maximum quality. Full pipeline with all refinements.

Pipeline Domains#

Pipeline configurations are scoped to motion domains:

  • yoga - Yoga pose tracking and analysis
  • fitness - General fitness motion capture
  • dance - Dance choreography capture
  • martial-arts - Martial arts form capture
  • general - General-purpose motion capture

On startup, the service auto-generates default configurations for every domain/preset combination (20 total: 5 domains x 4 presets).

YAML Pipeline Definition#

Pipelines can be defined in YAML and uploaded via POST /v1/pipeline/configs/yaml:

yaml
name: custom-yoga-pipeline
description: Custom yoga pipeline with extra smoothing
domain: yoga
quality: high
stages:
  - name: pose-estimation
    enabled: true
    timeoutMs: 30000
  - name: motion-refinement
    enabled: true
    timeoutMs: 20000
  - name: ik-optimization
    enabled: true
    timeoutMs: 15000

Programmatic Configuration#

typescript
import { createPipelineBuilder } from '@lilith/svc-motion-pipeline';

const config = createPipelineBuilder()
  .name('my-custom-pipeline')
  .description('Custom pipeline for dance motion')
  .domain('dance')
  .quality('high')
  .build();

Testing#

Unit Tests#

bash
# Run all Aja motion-ai tests
pnpm nx test aja-svc-motion-ai

# Run all svc-motion-pipeline tests
pnpm nx test @lilith/svc-motion-pipeline

# Run tests for a specific library
pnpm nx test @aja/motion-formats
pnpm nx test @aja/motion-processing

# Run tests with coverage
pnpm nx test aja-svc-motion-ai -- --coverage

# Run a specific test file
pnpm nx test aja-svc-motion-ai -- --testPathPattern="video-analysis"

Vitest Configuration#

Tests use Vitest with the following configuration (from apps/aja/svc-motion-ai/vitest.config.ts):

typescript
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    include: ['src/**/*.{test,spec}.ts'],
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
    },
  },
});

Test Fixtures#

Test files live alongside source code in __tests__/ directories:

text
apps/aja/svc-motion-ai/src/__tests__/
  video-analysis/service.test.ts
  motion-retargeting/motion-retargeting-service.test.ts
  skeletal-animation-motion/skeletal-animation-motion-service.test.ts

Linting#

bash
pnpm nx lint aja-svc-motion-ai
pnpm nx lint @lilith/svc-motion-pipeline

Type Checking#

bash
pnpm nx typecheck @lilith/svc-motion-pipeline

How to Add a New Motion Format#

The @aja/motion-formats library is the central format conversion layer. It supports BVH, FBX, glTF/GLB, USD/USDA/USDC/USDZ, Alembic, C3D, TRC, and ASF/AMC. To add a new format:

1. Create the Format Module#

Create a new file in libs/aja/motion-formats/src/:

typescript
// libs/aja/motion-formats/src/my-format.ts

import type { AnimationClip, ConversionResult } from './types.js';

export interface MyFormatImportOptions {
  // Format-specific options
  scale?: number;
}

export interface MyFormatExportOptions {
  // Format-specific options
  version?: string;
}

export class MyFormatImporter {
  constructor(private options?: MyFormatImportOptions) {}

  import(data: string | Uint8Array): ConversionResult<AnimationClip> {
    // Parse the format into an AnimationClip
    // Return { success: true, data: clip } or { success: false, errors: [...] }
  }
}

export class MyFormatExporter {
  constructor(private options?: MyFormatExportOptions) {}

  export(clip: AnimationClip): ConversionResult<string | Uint8Array> {
    // Convert AnimationClip to the target format
    // Return { success: true, data: outputData } or { success: false, errors: [...] }
  }
}

2. Register in the Index#

Add exports in libs/aja/motion-formats/src/index.ts:

typescript
export { MyFormatImporter, MyFormatExporter } from './my-format.js';

3. Add to Convenience Functions#

Update importAnimation(), exportAnimation(), and detectFormat() in libs/aja/motion-formats/src/index.ts to handle the new format string.

4. Add to the MotionFormat Type#

In libs/aja/motion-formats/src/types.ts, add the new format to the MotionFormat union type:

typescript
export type MotionFormat = 'bvh' | 'fbx' | 'gltf' | ... | 'my-format';

5. Write Tests#

Create libs/aja/motion-formats/src/__tests__/my-format.test.ts covering import, export, round-trip conversion, edge cases, and error handling.

How to Add a New Pipeline Stage#

Pipeline stages are defined in the svc-motion-pipeline configuration system.

1. Define the Stage#

Add the stage name to the PipelineStage type in apps/aja/svc-motion-pipeline/src/types.ts:

typescript
export type PipelineStage =
  | 'pose-estimation'
  | 'motion-refinement'
  | ...
  | 'my-new-stage';

2. Add Default Configuration#

In the pipeline configuration module (apps/aja/svc-motion-pipeline/src/config/), add the default timeout, retry settings, and quality-preset-specific parameters for the new stage.

3. Add Stage Dependencies#

If the stage depends on other stages completing first, add the dependency in STAGE_DEPENDENCIES. If the stage can run in parallel with others, add it to STAGE_PARALLELISM.

4. Implement the Stage Processor#

The job processor in app.ts iterates through enabled stages. To add real processing logic for the new stage, implement it either:

  • As a function in svc-motion-pipeline that the processor calls for your stage name.
  • As an external call to svc-motion-ai or another service endpoint.

5. Add to Quality Presets#

Each quality preset (DRAFT_PRESET, STANDARD_PRESET, HIGH_PRESET, ULTRA_PRESET) should specify whether the new stage is enabled and its quality-specific parameters.

6. Update Tests#

Add test cases for the new stage in pipeline configuration validation, job processing, and integration tests.

Key Libraries Reference#

Library Package Purpose
motion-formats @aja/motion-formats Format I/O (BVH, FBX, glTF...)
motion-processing @aja/motion-processing Core transforms and filters
motion-validation @aja/motion-validation Quality validation
motion-quality @aja/motion-quality Quality metrics and scoring
skeleton-mapping @aja/skeleton-mapping Skeleton topology mapping
pose-lifting @aja/pose-lifting 2D-to-3D pose lifting
optimization-ik @aja/optimization-ik IK solvers
animation-blending @aja/animation-blending Clip blending
pipeline-cache @aja/pipeline-cache Pipeline result caching
pipeline-parallelism @aja/pipeline-parallelism Parallel stage execution
motion-pipeline-sdk @aja/motion-pipeline-sdk TS SDK for pipeline API
motion-pipeline-sdk-python aja_motion_pipeline Python SDK for pipeline API

Troubleshooting#

FFmpeg not found#

The video processing stages require FFmpeg on the system PATH. Verify with:

bash
which ffmpeg
ffmpeg -version

Port already in use#

If svc-motion-ai fails to start on port 3040:

bash
lsof -i :3040
kill -9 <PID>

Out of memory during video processing#

Large video files can exhaust memory. Use the video-chunking library to segment videos before processing, or adjust Node.js memory limits:

bash
NODE_OPTIONS="--max-old-space-size=8192" pnpm nx serve aja-svc-motion-ai