# Aja Domain - Developer Guide

Aja is the motion capture and animation processing domain within the Oshun
monorepo. It provides services and libraries for video-to-motion extraction, AI
motion retargeting, skeletal animation, motion format conversion, and pipeline
orchestration.

## Architecture Overview

```
apps/aja/
  svc-motion-pipeline/   # Pipeline orchestration service (Fastify)
  svc-motion-ai/         # AI motion processing service (Hono)
  svc-reference-video/   # Reference video analysis service
  cli/                   # CLI tool for motion processing

libs/aja/
  motion-formats/        # BVH, FBX, glTF, USD, Alembic, C3D, TRC, ASF/AMC
  motion-processing/     # Core motion data transforms
  motion-validation/     # Quality validation and checking
  motion-quality/        # Quality metrics and scoring
  motion-pipeline-sdk/   # TypeScript SDK for pipeline API
  motion-pipeline-sdk-python/  # Python SDK for pipeline API
  motion-integration/    # Integration adapters
  skeleton-mapping/      # Skeleton topology mapping
  pose-lifting/          # 2D-to-3D pose lifting
  depth-sensing/         # Depth estimation
  human-mesh-recovery/   # Full body mesh from video
  neural-retargeting/    # Neural network motion retargeting
  semantic-retargeting/  # Semantic-aware motion retargeting
  blend-shape-retargeting/  # Facial blend shape retargeting
  proportional-adaptation/  # Body proportion adaptation
  optimization-ik/       # Inverse kinematics optimization
  animation-blending/    # Animation clip blending
  avatar-integration/    # Avatar system integration
  avatar-library/        # Reusable avatar models
  avatar-preview-ui/     # Avatar preview components
  bone-mapping-ui/       # Bone mapping editor UI
  domain-motion-pipelines/  # Domain-specific pipeline presets
  fitness-animation/     # Fitness-specific motion processing
  video-chunking/        # Video segmentation for processing
  multi-view-reconstruction/  # Multi-camera reconstruction
  batch-inference/       # Batch processing for ML models
  pipeline-cache/        # Pipeline result caching
  pipeline-parallelism/  # Parallel stage execution
  result-aggregation/    # Multi-output aggregation
  model-optimization/    # ML model optimization
  asset-storage/         # Motion file storage
  content-security/      # Content security scanning
  content-moderation/    # Content moderation
  content-watermarking/  # Watermarking for output files
  consent-management/    # Consent tracking for captured performers
  privacy-protection/    # PII detection/redaction in motion data
  data-retention/        # Data lifecycle management
```

## Prerequisites

| Tool    | Version | Purpose                             |
| ------- | ------- | ----------------------------------- |
| Node.js | >= 18.x | Runtime                             |
| pnpm    | >= 8.x  | Package manager                     |
| FFmpeg  | >= 5.x  | Video decoding, frame extraction    |
| Docker  | >= 24.x | Optional: containerized GPU workers |

Install FFmpeg (Ubuntu/Debian):

```bash
sudo apt update && sudo apt install -y ffmpeg
```

Verify:

```bash
ffmpeg -version
node --version
pnpm --version
```

Install monorepo dependencies from the repo root:

```bash
pnpm install
```

## Running Services Locally

### svc-motion-pipeline (Pipeline Orchestration)

The pipeline service is a Fastify-based HTTP API that manages motion processing
jobs: submission, configuration, status tracking, webhooks, SSE streaming, and
download.

```bash
# Development mode (auto-reload via tsx watch)
pnpm nx dev @lilith/svc-motion-pipeline

# Production build and start
pnpm nx build @lilith/svc-motion-pipeline
pnpm nx start @lilith/svc-motion-pipeline
```

The server starts on the port defined by `PORT` (default: inferred by
`@lilith/fastify-core`). Key environment variables:

| Variable       | Default           | Description        |
| -------------- | ----------------- | ------------------ |
| `SERVICE_NAME` | `motion-pipeline` | Service identifier |
| `PORT`         | (fastify-core)    | HTTP listen port   |
| `NODE_ENV`     | `development`     | Environment name   |

API routes (all under `/v1/pipeline/`):

| Method | Path                              | Purpose                      |
| ------ | --------------------------------- | ---------------------------- |
| GET    | `/health`                         | Health check                 |
| GET    | `/v1/pipeline/stats`              | Queue statistics             |
| GET    | `/v1/pipeline/configs`            | List pipeline configurations |
| GET    | `/v1/pipeline/configs/:id`        | Get config by ID             |
| GET    | `/v1/pipeline/configs/by-name/:n` | Get config by name           |
| POST   | `/v1/pipeline/configs`            | Create config (JSON)         |
| POST   | `/v1/pipeline/configs/yaml`       | Create config (YAML)         |
| GET    | `/v1/pipeline/configs/:id/yaml`   | Export config as YAML        |
| DELETE | `/v1/pipeline/configs/:id`        | Delete config                |
| GET    | `/v1/pipeline/presets`            | List quality presets         |
| POST   | `/v1/pipeline/jobs`               | Submit a job                 |
| GET    | `/v1/pipeline/jobs`               | List jobs                    |
| GET    | `/v1/pipeline/jobs/:id`           | Get job (polling)            |
| POST   | `/v1/pipeline/jobs/:id/cancel`    | Cancel a job                 |
| POST   | `/v1/pipeline/jobs/:id/retry`     | Retry a failed job           |
| PATCH  | `/v1/pipeline/jobs/:id/priority`  | Update job priority          |
| POST   | `/v1/pipeline/jobs/batch`         | Batch operations             |
| GET    | `/v1/pipeline/jobs/:id/outputs`   | List job outputs             |
| POST   | `/v1/pipeline/jobs/:id/download`  | Generate download URL        |
| GET    | `/v1/pipeline/jobs/:id/webhooks`  | Webhook deliveries for job   |
| GET    | `/v1/pipeline/jobs/:id/stream`    | SSE progress stream          |

### svc-motion-ai (AI Processing)

The motion AI service is a Hono-based HTTP API that runs the actual AI motion
processing: video-to-motion, retargeting, skeletal animation, and video
analysis.

```bash
# Development mode (auto-rebuild via nx serve)
pnpm nx serve aja-svc-motion-ai

# Run tests
pnpm nx test aja-svc-motion-ai
```

Default port: `3040` (configurable via `PORT` environment variable).

API routes:

| Method | Path                              | Purpose                   |
| ------ | --------------------------------- | ------------------------- |
| GET    | `/health`                         | Health check              |
| POST   | `/api/v1/video-to-motion/process` | Extract motion from video |
| POST   | `/api/v1/retargeting/retarget`    | Retarget motion data      |
| POST   | `/api/v1/skeletal/animate`        | Skeletal animation        |
| POST   | `/api/v1/analysis/analyze`        | Video analysis            |

Modules inside `svc-motion-ai`:

- **video-to-motion/** - Video preprocessing, pose estimation, temporal
  processing, keypoint normalization, and the main `VideoToMotionService`.
- **motion-retargeting/** - Motion retargeting between different skeleton
  topologies via `MotionRetargetingService`.
- **skeletal-animation-motion/** - Skeletal animation processing via
  `SkeletalAnimationMotionService`.
- **video-analysis/** - Video analysis via `VideoAnalysisService`.
- **motion-enhancement/** - Super-resolution, style transfer, synthesis, and
  physics refinement.
- **live-motion-capture/** - Real-time pose estimation, webcam capture, live
  avatar driving, and streaming pose lift.
- **mobile-live-capture/** - iOS/Android on-device capture and streaming.

### CLI

The Aja CLI provides command-line access to motion processing capabilities:

```bash
pnpm nx build aja-cli

# Available commands:
#   process   - Process a video through the pipeline
#   convert   - Convert between motion formats
#   inspect   - Inspect a motion file
#   jobs      - Manage pipeline jobs
#   health    - Check service health
#   config    - Manage pipeline configurations
#   debug     - Debug utilities
```

## Pipeline Configuration and Quality Presets

Pipeline configurations define which stages run, their parameters, quality
settings, input constraints, and output specifications. They can be created via
the JSON API, YAML endpoint, or the `PipelineBuilder` programmatic API.

### Quality Presets

Four built-in quality presets control the accuracy/speed trade-off:

| Preset     | Description                                          |
| ---------- | ---------------------------------------------------- |
| `draft`    | Fast processing, lower quality. Good for previews.   |
| `standard` | Balanced quality and speed. Default for most jobs.   |
| `high`     | Higher quality with more processing time.            |
| `ultra`    | Maximum quality. Full pipeline with all refinements. |

### Pipeline Domains

Pipeline configurations are scoped to motion domains:

- `yoga` - Yoga pose tracking and analysis
- `fitness` - General fitness motion capture
- `dance` - Dance choreography capture
- `martial-arts` - Martial arts form capture
- `general` - General-purpose motion capture

On startup, the service auto-generates default configurations for every
domain/preset combination (20 total: 5 domains x 4 presets).

### YAML Pipeline Definition

Pipelines can be defined in YAML and uploaded via
`POST /v1/pipeline/configs/yaml`:

```yaml
name: custom-yoga-pipeline
description: Custom yoga pipeline with extra smoothing
domain: yoga
quality: high
stages:
  - name: pose-estimation
    enabled: true
    timeoutMs: 30000
  - name: motion-refinement
    enabled: true
    timeoutMs: 20000
  - name: ik-optimization
    enabled: true
    timeoutMs: 15000
```

### Programmatic Configuration

```typescript
import { createPipelineBuilder } from '@lilith/svc-motion-pipeline';

const config = createPipelineBuilder()
  .name('my-custom-pipeline')
  .description('Custom pipeline for dance motion')
  .domain('dance')
  .quality('high')
  .build();
```

## Testing

### Unit Tests

```bash
# Run all Aja motion-ai tests
pnpm nx test aja-svc-motion-ai

# Run all svc-motion-pipeline tests
pnpm nx test @lilith/svc-motion-pipeline

# Run tests for a specific library
pnpm nx test @aja/motion-formats
pnpm nx test @aja/motion-processing

# Run tests with coverage
pnpm nx test aja-svc-motion-ai -- --coverage

# Run a specific test file
pnpm nx test aja-svc-motion-ai -- --testPathPattern="video-analysis"
```

### Vitest Configuration

Tests use Vitest with the following configuration (from
`apps/aja/svc-motion-ai/vitest.config.ts`):

```typescript
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    include: ['src/**/*.{test,spec}.ts'],
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
    },
  },
});
```

### Test Fixtures

Test files live alongside source code in `__tests__/` directories:

```
apps/aja/svc-motion-ai/src/__tests__/
  video-analysis/service.test.ts
  motion-retargeting/motion-retargeting-service.test.ts
  skeletal-animation-motion/skeletal-animation-motion-service.test.ts
```

### Linting

```bash
pnpm nx lint aja-svc-motion-ai
pnpm nx lint @lilith/svc-motion-pipeline
```

### Type Checking

```bash
pnpm nx typecheck @lilith/svc-motion-pipeline
```

## How to Add a New Motion Format

The `@aja/motion-formats` library is the central format conversion layer. It
supports BVH, FBX, glTF/GLB, USD/USDA/USDC/USDZ, Alembic, C3D, TRC, and ASF/AMC.
To add a new format:

### 1. Create the Format Module

Create a new file in `libs/aja/motion-formats/src/`:

```typescript
// libs/aja/motion-formats/src/my-format.ts

import type { AnimationClip, ConversionResult } from './types.js';

export interface MyFormatImportOptions {
  // Format-specific options
  scale?: number;
}

export interface MyFormatExportOptions {
  // Format-specific options
  version?: string;
}

export class MyFormatImporter {
  constructor(private options?: MyFormatImportOptions) {}

  import(data: string | Uint8Array): ConversionResult<AnimationClip> {
    // Parse the format into an AnimationClip
    // Return { success: true, data: clip } or { success: false, errors: [...] }
  }
}

export class MyFormatExporter {
  constructor(private options?: MyFormatExportOptions) {}

  export(clip: AnimationClip): ConversionResult<string | Uint8Array> {
    // Convert AnimationClip to the target format
    // Return { success: true, data: outputData } or { success: false, errors: [...] }
  }
}
```

### 2. Register in the Index

Add exports in `libs/aja/motion-formats/src/index.ts`:

```typescript
export { MyFormatImporter, MyFormatExporter } from './my-format.js';
```

### 3. Add to Convenience Functions

Update `importAnimation()`, `exportAnimation()`, and `detectFormat()` in
`libs/aja/motion-formats/src/index.ts` to handle the new format string.

### 4. Add to the MotionFormat Type

In `libs/aja/motion-formats/src/types.ts`, add the new format to the
`MotionFormat` union type:

```typescript
export type MotionFormat = 'bvh' | 'fbx' | 'gltf' | ... | 'my-format';
```

### 5. Write Tests

Create `libs/aja/motion-formats/src/__tests__/my-format.test.ts` covering
import, export, round-trip conversion, edge cases, and error handling.

## How to Add a New Pipeline Stage

Pipeline stages are defined in the `svc-motion-pipeline` configuration system.

### 1. Define the Stage

Add the stage name to the `PipelineStage` type in
`apps/aja/svc-motion-pipeline/src/types.ts`:

```typescript
export type PipelineStage =
  | 'pose-estimation'
  | 'motion-refinement'
  | ...
  | 'my-new-stage';
```

### 2. Add Default Configuration

In the pipeline configuration module
(`apps/aja/svc-motion-pipeline/src/config/`), add the default timeout, retry
settings, and quality-preset-specific parameters for the new stage.

### 3. Add Stage Dependencies

If the stage depends on other stages completing first, add the dependency in
`STAGE_DEPENDENCIES`. If the stage can run in parallel with others, add it to
`STAGE_PARALLELISM`.

### 4. Implement the Stage Processor

The job processor in `app.ts` iterates through enabled stages. To add real
processing logic for the new stage, implement it either:

- As a function in `svc-motion-pipeline` that the processor calls for your stage
  name.
- As an external call to `svc-motion-ai` or another service endpoint.

### 5. Add to Quality Presets

Each quality preset (`DRAFT_PRESET`, `STANDARD_PRESET`, `HIGH_PRESET`,
`ULTRA_PRESET`) should specify whether the new stage is enabled and its
quality-specific parameters.

### 6. Update Tests

Add test cases for the new stage in pipeline configuration validation, job
processing, and integration tests.

## Key Libraries Reference

| Library                      | Package                     | Purpose                        |
| ---------------------------- | --------------------------- | ------------------------------ |
| `motion-formats`             | `@aja/motion-formats`       | Format I/O (BVH, FBX, glTF...) |
| `motion-processing`          | `@aja/motion-processing`    | Core transforms and filters    |
| `motion-validation`          | `@aja/motion-validation`    | Quality validation             |
| `motion-quality`             | `@aja/motion-quality`       | Quality metrics and scoring    |
| `skeleton-mapping`           | `@aja/skeleton-mapping`     | Skeleton topology mapping      |
| `pose-lifting`               | `@aja/pose-lifting`         | 2D-to-3D pose lifting          |
| `optimization-ik`            | `@aja/optimization-ik`      | IK solvers                     |
| `animation-blending`         | `@aja/animation-blending`   | Clip blending                  |
| `pipeline-cache`             | `@aja/pipeline-cache`       | Pipeline result caching        |
| `pipeline-parallelism`       | `@aja/pipeline-parallelism` | Parallel stage execution       |
| `motion-pipeline-sdk`        | `@aja/motion-pipeline-sdk`  | TS SDK for pipeline API        |
| `motion-pipeline-sdk-python` | `aja_motion_pipeline`       | Python SDK for pipeline API    |

## Troubleshooting

### FFmpeg not found

The video processing stages require FFmpeg on the system PATH. Verify with:

```bash
which ffmpeg
ffmpeg -version
```

### Port already in use

If `svc-motion-ai` fails to start on port 3040:

```bash
lsof -i :3040
kill -9 <PID>
```

### Out of memory during video processing

Large video files can exhaust memory. Use the `video-chunking` library to
segment videos before processing, or adjust Node.js memory limits:

```bash
NODE_OPTIONS="--max-old-space-size=8192" pnpm nx serve aja-svc-motion-ai
```
