Aja Domain — Architecture

Aja — Motion AI and Animation Platform

Aja is a motion-intelligence platform that transforms raw video or motion capture input into clean, retargeted, production-ready animation data. Named after the Yoruba Orisha Aja — guide through the forest — the domain navigates the complex path from noisy, real-world capture to high-quality animation assets.

In practical terms, a caller submits a video file or a motion capture recording to Aja. The platform cleans and validates the data, estimates 3D skeletal pose, runs quality scoring, retargets the result to any target character skeleton, and delivers the finished animation in the requested file format. The same infrastructure that handles a single clip can scale out horizontally to process thousands of files in parallel. Downstream domains — game studios, VFX pipelines, AI coaching applications — consume the resulting BVH, FBX, or glTF files through integration adapters.

The domain consists of 5 application projects (apps/aja/) and 39 library projects (libs/aja/).

The motion-pipeline service, its SDK, and the motion-integration library were authored under the "Lilith Motion Pipeline" name and retain @lilith/... labels in code comments — and one package name (@lilith/svc-reference-video). The domain itself is Aja.

Library Organization#

The 39 libraries in libs/aja/ are organized into seven functional clusters. Each cluster below lists the libraries it contains and the concern they address.

libs/aja/
├── motion-formats/             # Animation format I/O (BVH, FBX, glTF/GLB, USD, Alembic, C3D, TRC, ASF/AMC, JSON)
├── motion-processing/          # Filtering, smoothing, interpolation, foot sliding correction
├── motion-quality/             # Quality metrics: smoothness, plausibility, ground-truth, perceptual
├── motion-validation/          # Data validation and error detection
├── motion-integration/         # Cross-domain integration adapters (Yemaya, Isis, Bellona, Sophia)
│
├── pose-lifting/               # 2D keypoints → 3D skeletal pose
├── depth-sensing/              # Monocular and stereo depth estimation
├── multi-view-reconstruction/  # Multi-camera 3D reconstruction
├── human-mesh-recovery/        # SMPL/SMPL-X parametric body mesh recovery
│
├── skeleton-mapping/           # Skeleton hierarchy conversion and normalization
├── optimization-ik/            # Inverse kinematics solvers for retargeting
├── avatar-library/             # Avatar definitions, hierarchy presets, rig configs
├── avatar-integration/         # Avatar binding and animation playback
├── avatar-preview-ui/          # React component for 3D avatar preview
├── bone-mapping-ui/            # React drag-and-drop bone mapping interface
│
├── semantic-retargeting/       # Intent-preserving motion transfer
├── proportional-adaptation/    # Body proportion adaptation during retargeting
├── blend-shape-retargeting/    # Facial blend shape extraction and remapping
├── neural-retargeting/         # ML-based motion retargeting
├── animation-blending/         # Motion blending, cross-fades, layering
│
├── domain-motion-pipelines/    # Pre-configured pipelines (yoga, fitness, dance, martial-arts)
├── fitness-animation/          # Fitness-specific analysis (reps, form, ROM)
├── film-pipeline/              # Film/VFX-grade motion deliverable packaging
│
├── motion-pipeline-sdk/        # TypeScript SDK for the pipeline service
├── motion-pipeline-sdk-python/ # Python SDK (aja-motion-pipeline)
├── batch-inference/            # Batch job orchestration
├── video-chunking/             # Automatic long-video splitting
├── model-optimization/         # Model compression and quantization
├── pipeline-parallelism/       # Multi-stage parallel processing
├── distributed-workers/        # Worker pool coordination
├── result-aggregation/         # Combine distributed results
├── pipeline-cache/             # LRU caching for pipeline stages
├── asset-storage/              # Tiered storage management, CDN delivery, signed URL access for motion assets
│
├── privacy-protection/         # Privacy-preserving processing
├── consent-management/         # Consent lifecycle tracking
├── content-security/           # Data security protocols
├── content-watermarking/       # Digital watermark embedding
├── content-moderation/         # Content screening
└── data-retention/             # Retention policy enforcement

Total: 39 library projects

Service Architecture#

Aja's runtime surface is two HTTP services, one library module, and a CLI tool. The table below lists each application project with its package name, runtime framework, default port, and core responsibility.

apps/aja/
  svc-motion-pipeline/    @aja/svc-motion-pipeline      Fastify   :8090   Core pipeline orchestration
  svc-motion-ai/          @aja/svc-motion-ai            Hono      :3040   Motion AI endpoints
  svc-reference-video/    @lilith/svc-reference-video   library   —       Reference video subsystem
  cli/                    @aja/cli                      Commander —       CLI tools
  docs/                   (unnamed)                     —         —       Markdown documentation site

There are 5 application projects. svc-reference-video ships as a TypeScript library (its package.json has main: dist/index.js and no HTTP server entry); docs is a static Markdown documentation tree.

svc-motion-pipeline (Fastify — port 8090)#

The central pipeline orchestrator. It is the entry point for all job submissions and the source of truth for job state. Its responsibilities are:

• Accept job submissions (single file, batch, URL)
• Manage the job queue (via Redis/BullMQ)
• Coordinate pipeline stage execution
• Deliver results and status updates
• Apply data retention and consent policies

svc-motion-ai (Hono — port 3040)#

The Motion AI service handles the inference-heavy operations that require GPU resources. It composes five internal services (VideoToMotionService, MotionRetargetingService, SkeletalAnimationMotionService, VideoAnalysisService, EmbodiedInstructionService) and exposes REST routes for:

• video-to-motion processing (POST /api/v1/video-to-motion/process)
• motion retargeting (POST /api/v1/retargeting/retarget)
• procedural skeletal animation (POST /api/v1/skeletal/animate)
• video analysis (POST /api/v1/analysis/analyze)
• embodied-instruction demonstration plans, coaching overlays, and session handoffs (POST /api/v1/embodied-instruction/*), plus a capabilities discovery endpoint

The service also contains internal module trees for live motion capture, mobile live capture, and motion enhancement (synthesis / style transfer / super-resolution / physics refinement) that the internal services use; these modules are not all directly exposed as routes. Live-capture streaming code lives in these modules — there is no live-capture route on the service.

svc-reference-video (@lilith/svc-reference-video — library)#

A library module (not a server) that manages a searchable collection of reference videos used for comparison, benchmarking, and instructional content. Its responsibilities are:

• Ingest reference videos (upload, URL, YouTube, Vimeo)
• Scene detection, metadata extraction, person/activity detection
• Categorization, tagging, and segmentation
• Search over the reference library (CLIP-embedding, Meilisearch, and Qdrant backends are present in tests)
• Annotation, collaboration (workspaces, comments, reviews, assignments), and version history

cli (Commander)#

Command-line tools for developers and pipeline operators. The CLI wraps the TypeScript SDK (@aja/motion-pipeline-sdk) and exposes aja process, aja convert, aja inspect, aja debug, aja config, aja jobs, and aja health commands.

Processing Pipeline Architecture#

The diagram below maps each pipeline stage to the library that implements it, showing how data flows from raw input through to a finished animation file.

1. INPUT STAGE
   ├── Video upload (file, URL, stream)
   ├── MoCap import (C3D, TRC, BVH, FBX, ASF/AMC)
   └── Live camera feed (WebSocket)
         │
         ▼
2. VIDEO ANALYSIS STAGE
   ├── @aja/pose-lifting         2D → 3D pose
   ├── @aja/depth-sensing        Depth from mono/stereo
   ├── @aja/multi-view-reconstruction  Multi-camera 3D
   └── @aja/human-mesh-recovery  SMPL-X full body mesh
         │
         ▼
3. MOTION PROCESSING STAGE
   ├── @aja/motion-processing    Filter, smooth, interpolate
   ├── @aja/motion-validation    Validate data integrity
   └── @aja/motion-quality       Score quality dimensions
         │
         ▼
4. RETARGETING STAGE
   ├── @aja/skeleton-mapping     Map bone hierarchies
   ├── @aja/proportional-adaptation  Adapt body proportions
   ├── @aja/semantic-retargeting     Intent-preserving transfer
   ├── @aja/neural-retargeting       ML-based transfer
   ├── @aja/blend-shape-retargeting  Facial blend shapes
   └── @aja/animation-blending       Blend and transition
         │
         ▼
5. EXPORT STAGE
   └── @aja/motion-formats       Write to BVH, FBX, GLB, USD, ...

The capability map above illustrates the full potential of the pipeline. The orchestrated pipeline in svc-motion-pipeline uses a concrete PipelineStage enum of 11 stages: ingestion, validation, preprocessing, pose-estimation, skeleton-fitting, domain-analysis, quality-assessment, retargeting, format-conversion, postprocessing, delivery.

The V1 job processor (createPipelineProcessor, task V1-P2-1775) runs ingestion, validation, preprocessing, quality-assessment, format-conversion, postprocessing, and delivery using @aja/motion-formats. The four ML-only stages — pose-estimation, skeleton-fitting, domain-analysis, retargeting — are V2-deferred per descope decision V1-P2-0331; when a pipeline config enables them the processor marks each 'skipped' with a structured log line and continues.

Core Design Patterns#

Five architectural decisions shape how every library in Aja is built. These patterns enforce consistency, enable horizontal scaling, and protect user data.

1. Stage-Based Pipeline with Caching#

The pipeline is broken into discrete, cacheable stages. Each stage has a well-defined input and output type. @aja/pipeline-cache caches stage outputs using a content hash key, so a second job with the same input video does not repeat expensive pose estimation. Stages are designed to be independently testable and replaceable.

2. Quality-Gated Output#

Every pipeline run produces a QualityReport alongside the motion output. Applications consuming Aja can configure a minimum quality threshold — jobs below the threshold are flagged for manual review rather than automatically delivered. Quality scoring happens as a dedicated stage after processing and before retargeting, allowing early detection of inputs unlikely to produce usable output.

3. Format Universality#

@aja/motion-formats is the single source of truth for all format I/O. The library implements a common internal representation (MotionClip) that all other libraries work with, and handles serialization to/from all supported formats. This prevents format-specific code from leaking into processing or retargeting libraries.

4. Distributed Horizontal Scaling#

Heavy AI workloads are distributed via @aja/distributed-workers and @aja/pipeline-parallelism. Worker pools can be scaled horizontally on Kubernetes. @aja/video-chunking ensures long videos are split into processable segments before dispatch, enabling each chunk to be processed independently and in parallel. @aja/result-aggregation assembles the per-chunk results into a coherent output clip.

5. Privacy by Design#

@aja/privacy-protection is the first output consumer in the pipeline. When privacyMode is enabled, the source video frames are deleted from storage immediately after pose extraction completes — before any other output is written. Only anonymized skeletal data flows downstream. @aja/consent-management records and enforces consent per subject, allowing withdrawn consent to trigger retroactive data deletion.

Library Dependency Graph (Key Paths)#

Understanding which libraries depend on which others is important for making changes safely. The graph below shows the most important dependency paths, derived from the dependencies declared in each library's package.json. @aja/motion-formats is the foundation: it has no internal Aja dependencies and everything else ultimately builds on top of it.

@aja/motion-pipeline-sdk
  └── uuid, eventsource          (no @aja deps; calls svc-motion-pipeline via HTTP)

@aja/neural-retargeting
  ├── @aja/skeleton-mapping
  └── @aja/optimization-ik

@aja/skeleton-mapping
  └── @aja/motion-formats        (re-exports core primitives)

@aja/motion-quality
  └── @aja/motion-formats        (re-exports core primitives)

@aja/domain-motion-pipelines
  └── @oshun/types               (defines its own motion types)

@aja/film-pipeline
  └── @aja/motion-formats        (type-only import of AnimationClip / MotionFormat)

@aja/human-mesh-recovery
  └── tslib                      (no @aja deps)

apps/aja/svc-motion-ai
  └── @aja/{motion-formats, motion-processing, motion-quality, skeleton-mapping,
      optimization-ik, pose-lifting, depth-sensing, human-mesh-recovery,
      multi-view-reconstruction, neural-retargeting, domain-motion-pipelines}

apps/aja/svc-motion-pipeline
  └── @aja/{domain-motion-pipelines, motion-formats, motion-processing,
      motion-quality, motion-validation}

@aja/motion-formats   (no internal dependencies — foundation library)

Dependencies on Other Oshun Domains#

Aja is a producer domain: it consumes shared infrastructure from @oshun/* but does not depend on any other Oshun product domain. Instead, downstream product domains pull motion data from Aja when they need it, decoupling their release cycles from Aja's pipeline changes.

The integration boundary is implemented in @aja/motion-integration, which ships adapter modules for exactly four domains. Each adapter translates Aja's AnimationClip and quality data into the format and protocol expected by the receiving domain. The IntegrationDomain type is 'yemaya' | 'isis' | 'bellona' | 'sophia', and each has a dedicated adapter under motion-integration/src/{domain}/.

The Yemaya boundary exists because film production requires Alembic or FBX deliverables formatted for DCC tools (Maya, Houdini, Nuke) with a production session context — a concern that belongs to Yemaya, not to Aja's core pipeline. The Isis boundary keeps 3D asset assembly separate from motion processing: Isis receives the finished animation paired with a mesh and integrates it into its asset library. The Bellona boundary enables deterministic build pipelines — motion artifacts are versioned and published to Bellona's registry so that dependent builds are reproducible.

Consuming Domains#

Aja is a producer domain — it processes raw input into standardized motion assets consumed by downstream domains. The Yemaya remote-film-capture library is a verified consumer: it imports Aja libraries directly (aja-retargeting-integration.ts, aja-film-pipeline-integration.ts, aja-multi-view-integration.ts, and others under libs/yemaya/remote-film-capture/src/). Aja does not depend on other Oshun product domains, only on shared infrastructure.

Build and Test Configuration#

All Aja projects share the same build toolchain and tag conventions. The key executor values and direct invocations for use when Nx is unavailable (e.g., due to worktree conflicts) are below.

• Build executor: @nx/js:tsc
• Lint executor: @nx/eslint:lint
• Test executor: @nx/vite:test (Vitest for TypeScript)
• Tags: scope:aja, layer:domain, type:lib

Library project.json name values carry a lilith- prefix (e.g. lilith-motion-formats) reflecting the original "Lilith Motion Pipeline" authorship; the published package names are @aja/....

Direct invocations when Nx is unavailable:

# Type check
cd libs/aja/<library> && npx tsc --noEmit

# Run TypeScript tests
cd libs/aja/<library> && npx vitest run

# Start the pipeline service directly (entry: src/server.ts)
cd apps/aja/svc-motion-pipeline && npx tsx src/server.ts

# Start the AI service directly (entry: src/server.ts)
cd apps/aja/svc-motion-ai && npx tsx src/server.ts

Source Verification#

This architecture document was checked against the source under apps/aja/* and libs/aja/*: 39 library package.json files, the 5 app projects, the service entry points (server.ts), the PipelineStage enum and V2_DEFERRED_STAGES set in svc-motion-pipeline, the motion-integration IntegrationDomain type, and the verified Yemaya consumer under libs/yemaya/remote-film-capture/. Project / package-name discrepancies (@lilith/... labels) are noted where they occur.