# Pipeline Architecture

Comprehensive architecture diagrams for all Yemaya platform pipelines.

## Table of Contents

1. [System Overview](#system-overview)
2. [Game Creation Pipeline](#game-creation-pipeline)
3. [Film Creation Pipeline](#film-creation-pipeline)
4. [Pipeline Infrastructure](#pipeline-infrastructure)
5. [Quality Gates System](#quality-gates-system)
6. [Worker Queue System](#worker-queue-system)
7. [3D Generation Pipeline](#3d-generation-pipeline)
8. [Agent Orchestration System](#agent-orchestration-system)
9. [Data Flow Patterns](#data-flow-patterns)

---

## System Overview

### High-Level Architecture

```mermaid
flowchart TB
    subgraph Clients["Client Applications"]
        Web["Web App"]
        Desktop["Desktop App"]
        CLI["CLI Tool"]
        SDK["TypeScript/Python SDK"]
    end

    subgraph API["API Layer"]
        REST["REST API<br/>(Hono)"]
        WS["WebSocket<br/>(Real-time)"]
        GRPC["gRPC Services<br/>(Protobuf)"]
    end

    subgraph Pipelines["Pipeline Orchestration"]
        GamePipe["Game Creation<br/>Pipeline"]
        FilmPipe["Film Creation<br/>Pipeline"]
        ThreeD["3D Generation<br/>Pipeline"]
        Audio["Audio Pipeline"]
    end

    subgraph Orchestration["Agent Orchestration"]
        Crew["Crew System"]
        Workflow["Workflow Engine"]
        Delegation["Task Delegation"]
        RAG["RAG Pipeline"]
    end

    subgraph Workers["Worker System (BullMQ)"]
        AgentWorker["Agent Worker"]
        MediaWorker["Media Worker"]
        AssetWorker["Asset Worker"]
        Gen3DWorker["3D Gen Worker"]
        NotifyWorker["Notification Worker"]
    end

    subgraph QualityGates["Quality Gates"]
        Narrative["Narrative QG"]
        Visual["Visual QG"]
        AudioQG["Audio QG"]
        Technical["Technical QG"]
        Gameplay["Gameplay QG"]
    end

    subgraph Infrastructure["Infrastructure"]
        Postgres[(PostgreSQL)]
        Redis[(Redis)]
        MinIO[(MinIO/S3)]
        CDN["CDN"]
    end

    subgraph External["External Services"]
        AI["AI Providers<br/>(Claude, OpenAI, Gemini)"]
        ThreeDProviders["3D Providers<br/>(Rodin, Meshy, etc.)"]
        GameEngines["Game Engines<br/>(Blender, Godot, Unreal)"]
    end

    Clients --> API
    API --> Pipelines
    Pipelines --> Orchestration
    Pipelines --> Workers
    Pipelines --> QualityGates
    Workers --> Infrastructure
    Workers --> External
    Orchestration --> AI
    ThreeD --> ThreeDProviders
    GamePipe --> GameEngines
    FilmPipe --> GameEngines
```

### Pipeline Types Comparison

```mermaid
graph LR
    subgraph Types["Pipeline Types"]
        direction TB
        Game["Game Pipeline<br/>8 Phases<br/>3,336 LOC"]
        Film["Film Pipeline<br/>5 Phases<br/>2,345 LOC"]
        Infra["Infrastructure<br/>4 Components<br/>1,716 LOC"]
        QG["Quality Gates<br/>5 Types<br/>3,033 LOC"]
        Test["Automated Testing<br/>4 Categories<br/>2,154 LOC"]
    end

    Game --> |"Concept → Release"| GameOut["Playable Game"]
    Film --> |"Development → Delivery"| FilmOut["Final Film"]
    Infra --> |"Execution Support"| InfraOut["Pipeline Ops"]
    QG --> |"Validation"| QGOut["Quality Assurance"]
    Test --> |"Verification"| TestOut["Test Reports"]
```

---

## Game Creation Pipeline

### Complete Phase Flow

```mermaid
flowchart TB
    subgraph Phase1["Phase 1: Concept Generation"]
        UserPrompt["User Prompt"]
        GameConcept["Game Concept"]
        ArtDirection["Art Direction"]
        MarketAnalysis["Market Analysis"]
        MoodBoards["Mood Boards"]
    end

    subgraph Phase2["Phase 2: GDD Generation"]
        CoreLoop["Core Loop Design"]
        Mechanics["Mechanic Design"]
        Systems["System Design"]
        Levels["Level Design"]
        Narrative["Narrative Design"]
    end

    subgraph Phase3["Phase 3: Prototype"]
        Prototype["Playable Prototype"]
        Metrics["Prototype Metrics"]
        Feedback["Playtester Feedback"]
        Iteration["Prototype Iteration"]
    end

    subgraph Phase4["Phase 4: Asset Production"]
        AssetPlan["Asset Production Plan"]
        Models["3D Models"]
        Animations["Animations"]
        Textures["Textures"]
        VFX["Visual Effects"]
        Audio["Audio Assets"]
    end

    subgraph Phase5["Phase 5: Level Assembly"]
        LevelAssembly["Level Construction"]
        Scripting["Gameplay Scripting"]
        Triggers["Trigger Setup"]
        Lighting["Lighting Setup"]
        UI["UI Implementation"]
    end

    subgraph Phase6["Phase 6: Polish & Optimization"]
        BugFix["Bug Fixing"]
        PerfOpt["Performance Optimization"]
        Balance["Balance Tuning"]
        Juice["Juice Pass"]
        Accessibility["Accessibility"]
    end

    subgraph Phase7["Phase 7: QA & Testing"]
        TestStrategy["Testing Strategy"]
        AutoTests["Automated Tests"]
        AIPlaytest["AI Playtesting"]
        ManualTest["Manual Testing"]
    end

    subgraph Phase8["Phase 8: Release"]
        PlatformBuild["Platform Builds"]
        StoreSubmit["Store Submission"]
        Marketing["Marketing"]
        Launch["Launch"]
    end

    Phase1 --> |QualityGate| Phase2
    Phase2 --> |QualityGate| Phase3
    Phase3 --> |QualityGate| Phase4
    Phase4 --> |QualityGate| Phase5
    Phase5 --> |QualityGate| Phase6
    Phase6 --> |QualityGate| Phase7
    Phase7 --> |QualityGate| Phase8
    Phase8 --> Released["Released Game"]

    style Phase1 fill:#e1f5fe
    style Phase2 fill:#e8f5e9
    style Phase3 fill:#fff3e0
    style Phase4 fill:#fce4ec
    style Phase5 fill:#f3e5f5
    style Phase6 fill:#e0f2f1
    style Phase7 fill:#fff8e1
    style Phase8 fill:#efebe9
```

### Asset Production Parallel Flow

```mermaid
flowchart LR
    AssetSpec["Asset Specification<br/>(from GDD)"]

    subgraph Parallel["Parallel Production"]
        direction TB

        subgraph ModelPipe["3D Model Pipeline"]
            Generate3D["Generate 3D"]
            MeshProcess["Mesh Processing"]
            LODGen["LOD Generation"]
        end

        subgraph TexturePipe["Texture Pipeline"]
            GenTexture["Generate Textures"]
            PBRSetup["PBR Setup"]
            BakeMaps["Bake Maps"]
        end

        subgraph AnimPipe["Animation Pipeline"]
            Rigging["Auto-Rigging"]
            AnimGen["Animation Gen"]
            Retarget["Retargeting"]
        end

        subgraph AudioPipe["Audio Pipeline"]
            SFXGen["SFX Generation"]
            MusicGen["Music Generation"]
            VoiceSynth["Voice Synthesis"]
        end
    end

    AssetSpec --> ModelPipe
    AssetSpec --> TexturePipe
    AssetSpec --> AnimPipe
    AssetSpec --> AudioPipe

    ModelPipe --> Merge["Asset Merge"]
    TexturePipe --> Merge
    AnimPipe --> Merge
    AudioPipe --> Merge

    Merge --> Validate["Validation"]
    Validate --> Export["Export to Engine"]
```

---

## Film Creation Pipeline

### Complete Phase Flow

```mermaid
flowchart TB
    subgraph Dev["Phase 1: Development"]
        Concept["Film Concept"]
        Treatment["Treatment"]
        PitchDeck["Pitch Deck"]
        Budget["Budget Estimate"]
        Casting["Casting Suggestions"]
        Locations["Location Suggestions"]
    end

    subgraph PreProd["Phase 2: Pre-Production"]
        Screenplay["Screenplay"]
        Storyboard["Storyboards"]
        ShotList["Shot Lists"]
        Schedule["Production Schedule"]
        BudgetBreak["Budget Breakdown"]
        CallSheets["Call Sheets"]
    end

    subgraph Prod["Phase 3: Animation Production"]
        Environments["Environment Assets"]
        Characters["Character Assets"]
        Props["Prop Assets"]
        Animation["Animation"]
        LightingSetup["Lighting Setup"]
        Rendering["Rendering"]
    end

    subgraph Post["Phase 4: Post-Production"]
        EditAssembly["Edit Assembly"]
        VFX["VFX Integration"]
        ColorGrade["Color Grading"]
        SoundDesign["Sound Design"]
        MusicScore["Music Scoring"]
        FinalMix["Final Mix"]
    end

    subgraph Delivery["Phase 5: Delivery"]
        DCP["DCP Creation"]
        Transcode["Transcoding"]
        QC["Quality Control"]
        Distribute["Distribution"]
        Subtitles["Subtitles"]
        Dubbing["Dubbing"]
    end

    Dev --> |Greenlight| PreProd
    PreProd --> |Approved| Prod
    Prod --> |Rendered| Post
    Post --> |Final| Delivery
    Delivery --> Released["Released Film"]

    style Dev fill:#e3f2fd
    style PreProd fill:#e8f5e9
    style Prod fill:#fff3e0
    style Post fill:#fce4ec
    style Delivery fill:#f3e5f5
```

### Post-Production Parallel Workflow

```mermaid
flowchart TB
    RenderedShots["Rendered Shots"]

    subgraph PostParallel["Parallel Post-Production"]
        direction LR

        subgraph EditTrack["Edit Track"]
            RoughCut["Rough Cut"]
            FineCut["Fine Cut"]
            FinalCut["Final Cut"]
        end

        subgraph VFXTrack["VFX Track"]
            VFXPlanning["VFX Planning"]
            VFXExec["VFX Execution"]
            VFXComp["Compositing"]
        end

        subgraph AudioTrack["Audio Track"]
            Dialogue["Dialogue Edit"]
            Foley["Foley"]
            Ambience["Ambience"]
        end

        subgraph MusicTrack["Music Track"]
            Temp["Temp Score"]
            Compose["Composition"]
            Record["Recording"]
        end
    end

    RenderedShots --> PostParallel

    PostParallel --> ColorGrade["Color Grading"]
    ColorGrade --> FinalMix["Final Mix"]
    FinalMix --> Master["Master Delivery"]
```

---

## Pipeline Infrastructure

### Executor Architecture

```mermaid
flowchart TB
    subgraph Executor["PipelineExecutor"]
        PhaseQueue["Phase Queue"]

        subgraph Execution["Execution Engine"]
            StateManager["State Manager"]
            CheckpointMgr["Checkpoint Manager"]
            RetryPolicy["Retry Policy"]
            RollbackMgr["Rollback Manager"]
        end

        subgraph ResourceMgmt["Resource Management"]
            Scheduler["Resource Scheduler"]
            Allocator["Resource Allocator"]
            CostTracker["Cost Tracker"]
            Utilization["Utilization Monitor"]
        end

        subgraph Monitoring["Monitoring"]
            Monitor["Pipeline Monitor"]
            Alerts["Alert System"]
            Dashboard["Dashboard"]
            Analytics["Analytics"]
        end
    end

    Input["Phase Definition"] --> PhaseQueue
    PhaseQueue --> Execution
    Execution --> ResourceMgmt
    Execution --> Monitoring

    Execution --> |Success| Output["Phase Output"]
    Execution --> |Failure| Recovery["Recovery/Rollback"]

    style Executor fill:#e3f2fd
```

### Retry Policy Strategies

```mermaid
flowchart LR
    subgraph Strategies["Retry Strategies"]
        direction TB

        Linear["Linear Backoff<br/>delay × attempt"]
        Exponential["Exponential Backoff<br/>delay × 2^attempt"]
        Fixed["Fixed Delay<br/>constant delay"]
    end

    Failure["Phase Failure"] --> Check{"Attempts<br/>Remaining?"}
    Check --> |Yes| Strategies
    Check --> |No| Final["Final Failure"]

    Strategies --> Wait["Wait Period"]
    Wait --> Retry["Retry Execution"]
    Retry --> |Success| Continue["Continue Pipeline"]
    Retry --> |Failure| Check
```

### Template System

```mermaid
flowchart TB
    subgraph Templates["Pipeline Templates"]
        direction LR
        GameTemplate["Game Template"]
        FilmTemplate["Film Template"]
        CustomTemplate["Custom Template"]
    end

    subgraph Variables["Template Variables"]
        ProjectName["Project Name"]
        Genre["Genre"]
        Platform["Target Platform"]
        Quality["Quality Level"]
    end

    subgraph Overrides["Phase Overrides"]
        SkipPhase["Skip Phases"]
        CustomPhase["Custom Phases"]
        ModifyPhase["Modified Phases"]
    end

    Templates --> Instantiate["Template Instantiation"]
    Variables --> Instantiate
    Overrides --> Instantiate

    Instantiate --> Pipeline["Configured Pipeline"]
    Pipeline --> Execute["Execute"]
```

---

## Quality Gates System

### Quality Gate Flow

```mermaid
flowchart TB
    PhaseOutput["Phase Output"]

    subgraph Assessment["Quality Assessment"]
        Collect["Collect Artifacts"]
        Analyze["Run Analysis"]
        Score["Calculate Scores"]
    end

    PhaseOutput --> Assessment

    Assessment --> Decision{"Quality<br/>Decision"}

    Decision --> |"Score >= Threshold"| Pass["PASS"]
    Decision --> |"Score < Threshold"| Fail["FAIL"]
    Decision --> |"Manual Review"| Override["OVERRIDE"]

    Pass --> Continue["Continue Pipeline"]

    Fail --> Remediate["Remediation Phase"]
    Remediate --> |"Fixes Applied"| Assessment
    Remediate --> |"Cannot Fix"| Block["Block Pipeline"]

    Override --> |"With Justification"| Continue
    Override --> |"Rejected"| Block

    style Pass fill:#c8e6c9
    style Fail fill:#ffcdd2
    style Override fill:#fff9c4
```

### Quality Gate Types

```mermaid
flowchart LR
    subgraph NarrativeGate["Narrative Quality Gate"]
        direction TB
        ScriptQuality["Script Quality"]
        DialogueCheck["Dialogue Check"]
        CharConsist["Character Consistency"]
        PlotCoherence["Plot Coherence"]
    end

    subgraph VisualGate["Visual Quality Gate"]
        direction TB
        StyleConsist["Style Consistency"]
        TechSpecs["Technical Specs"]
        ColorPalette["Color Palette"]
        Optimization["Optimization"]
    end

    subgraph AudioGate["Audio Quality Gate"]
        direction TB
        LevelCheck["Level Check"]
        FreqResponse["Frequency Response"]
        SyncCheck["Sync Verification"]
        FormatCheck["Format Compliance"]
    end

    subgraph TechnicalGate["Technical Quality Gate"]
        direction TB
        Performance["Performance"]
        Stability["Stability"]
        Compatibility["Compatibility"]
        Memory["Memory Usage"]
    end

    subgraph GameplayGate["Gameplay Quality Gate"]
        direction TB
        Balance["Balance"]
        Difficulty["Difficulty Curve"]
        FunFactor["Fun Factor"]
        Pacing["Pacing"]
    end

    Input["Phase Output"] --> NarrativeGate
    Input --> VisualGate
    Input --> AudioGate
    Input --> TechnicalGate
    Input --> GameplayGate

    NarrativeGate --> Aggregate["Aggregate Score"]
    VisualGate --> Aggregate
    AudioGate --> Aggregate
    TechnicalGate --> Aggregate
    GameplayGate --> Aggregate

    Aggregate --> Final["Quality Decision"]
```

### Issue Classification

```mermaid
flowchart TB
    subgraph IssueTypes["Issue Types"]
        direction LR

        subgraph Critical["Critical"]
            Crash["Crashes"]
            DataLoss["Data Loss"]
            Security["Security Issues"]
        end

        subgraph Major["Major"]
            Broken["Broken Features"]
            PerfIssue["Performance Issues"]
            Regression["Regressions"]
        end

        subgraph Minor["Minor"]
            Visual["Visual Glitches"]
            Polish["Polish Issues"]
            Edge["Edge Cases"]
        end
    end

    Critical --> |"Must Fix"| Block["Block Release"]
    Major --> |"Should Fix"| Remediate["Remediation Required"]
    Minor --> |"Can Ship"| Track["Track for Future"]

    style Critical fill:#ffcdd2
    style Major fill:#fff9c4
    style Minor fill:#c8e6c9
```

---

## Worker Queue System

### BullMQ Architecture

```mermaid
flowchart TB
    subgraph API["API Layer"]
        REST["REST Endpoints"]
        GRPC["gRPC Services"]
        WS["WebSocket"]
    end

    subgraph Queues["BullMQ Queues"]
        AgentQ["yemaya:agent"]
        MediaQ["yemaya:media"]
        AssetQ["yemaya:asset"]
        Gen3DQ["yemaya:generation-3d"]
        NotifyQ["yemaya:notification"]
    end

    subgraph Workers["Worker Processes"]
        AgentW["Agent Worker<br/>3 retries, exponential"]
        MediaW["Media Worker<br/>2 retries, fixed"]
        AssetW["Asset Worker<br/>3 retries"]
        Gen3DW["3D Gen Worker<br/>3 retries, 7-day retention"]
        NotifyW["Notification Worker<br/>5 retries"]
    end

    subgraph Backend["Queue Backend"]
        Redis[(Redis)]
    end

    API --> |"Add Job"| Queues
    Queues --> Redis
    Redis --> Workers

    Workers --> |"Job Complete"| Events["Event Emission"]
    Workers --> |"Progress"| WS
    Events --> |"Webhook"| External["External Systems"]

    style Redis fill:#ffcdd2
```

### Job Type Details

```mermaid
flowchart LR
    subgraph AgentJobs["Agent Jobs"]
        direction TB
        ExecuteTask["agent.execute-task"]
        BatchProcess["agent.batch-process"]
        GenScript["agent.generate-script"]
        AnalyzeContent["agent.analyze-content"]
        Collaborate["agent.collaborate"]
    end

    subgraph MediaJobs["Media Jobs"]
        direction TB
        TranscodeV["media.transcode-video"]
        TranscodeA["media.transcode-audio"]
        ExtractFrames["media.extract-frames"]
        GenThumb["media.generate-thumbnail"]
        Concatenate["media.concatenate"]
    end

    subgraph AssetJobs["Asset Jobs"]
        direction TB
        OptimizeImg["asset.optimize-image"]
        GenVariants["asset.generate-variants"]
        ExtractMeta["asset.extract-metadata"]
        Validate["asset.validate"]
    end

    subgraph Gen3DJobs["3D Generation Jobs"]
        direction TB
        TextTo3D["generation-3d.text-to-3d"]
        ImageTo3D["generation-3d.image-to-3d"]
        MeshProcess["generation-3d.mesh-process"]
        AutoRig["generation-3d.auto-rig"]
        TextureGen["generation-3d.texture-generate"]
    end
```

### Job Lifecycle

```mermaid
stateDiagram-v2
    [*] --> Waiting: Job Added
    Waiting --> Active: Worker Claims
    Active --> Completed: Success
    Active --> Failed: Error
    Failed --> Waiting: Retry
    Failed --> [*]: Max Retries
    Completed --> [*]: Done

    Active --> Delayed: Rate Limited
    Delayed --> Waiting: Delay Expires
```

---

## 3D Generation Pipeline

### Provider Architecture

```mermaid
flowchart TB
    subgraph Input["Input Types"]
        TextPrompt["Text Prompt"]
        Image["Reference Image"]
        MultiView["Multi-View Images"]
    end

    subgraph Providers["3D Generation Providers"]
        Rodin["Rodin"]
        Meshy["Meshy"]
        Tripo["Tripo"]
        Trellis["Trellis"]
        Hunyuan["Hunyuan"]
        Threedfy["Threedfy"]
        Local["Local Model"]
    end

    subgraph Processing["Mesh Processing"]
        Optimize["Optimization"]
        Cleanup["Mesh Cleanup"]
        UVUnwrap["UV Unwrapping"]
        LODGen["LOD Generation"]
    end

    subgraph Texturing["Texture Processing"]
        PBRGen["PBR Generation"]
        TextureApply["Texture Application"]
        BakeMaps["Map Baking"]
    end

    subgraph Export["Export Formats"]
        GLB["GLB"]
        GLTF["GLTF"]
        FBX["FBX"]
        OBJ["OBJ"]
        USD["USD/USDZ"]
    end

    Input --> |"Provider Selection"| Providers
    Providers --> |"Raw Mesh"| Processing
    Processing --> |"Clean Mesh"| Texturing
    Texturing --> |"Textured Model"| Export

    style Providers fill:#e3f2fd
    style Processing fill:#e8f5e9
    style Texturing fill:#fff3e0
```

### Quality Levels

```mermaid
flowchart LR
    subgraph QualityLevels["Quality Levels"]
        Draft["Draft<br/>Low poly, fast"]
        Standard["Standard<br/>Balanced"]
        High["High<br/>Detailed"]
        Ultra["Ultra<br/>Maximum quality"]
    end

    Draft --> |"~30s"| DraftOut["Preview Model"]
    Standard --> |"~2min"| StdOut["Production Ready"]
    High --> |"~5min"| HighOut["High Detail"]
    Ultra --> |"~10min"| UltraOut["Hero Asset"]

    style Draft fill:#fff9c4
    style Standard fill:#c8e6c9
    style High fill:#bbdefb
    style Ultra fill:#e1bee7
```

### Complete Generation Flow

```mermaid
sequenceDiagram
    participant Client
    participant API
    participant Queue
    participant Worker
    participant Provider
    participant Storage

    Client->>API: POST /3d/generate
    API->>Queue: Add Job
    API-->>Client: { jobId, status: "queued" }

    Queue->>Worker: Process Job
    Worker->>Provider: Generate 3D Model

    loop Progress Updates
        Provider-->>Worker: Progress %
        Worker-->>API: WebSocket Update
        API-->>Client: Progress Event
    end

    Provider-->>Worker: Raw Model
    Worker->>Worker: Mesh Processing
    Worker->>Worker: Texture Processing
    Worker->>Storage: Upload Assets
    Worker->>API: Job Complete
    API-->>Client: Completion Event

    Client->>API: GET /3d/jobs/{id}
    API-->>Client: { status: "completed", assets: [...] }
```

---

## Agent Orchestration System

### Crew System Architecture

```mermaid
flowchart TB
    subgraph CrewDefinition["Crew Definition"]
        Roles["Agent Roles"]
        Goals["Role Goals"]
        Tools["Available Tools"]
        Backstory["Agent Backstory"]
    end

    subgraph TaskManagement["Task Management"]
        Tasks["Task Queue"]
        Dependencies["Task Dependencies"]
        Priority["Priority Assignment"]
    end

    subgraph ExecutionModes["Execution Modes"]
        Sequential["Sequential<br/>One-by-one"]
        Parallel["Parallel<br/>Concurrent"]
        Hierarchical["Hierarchical<br/>Manager delegates"]
    end

    subgraph Communication["Communication"]
        Bus["Crew Message Bus"]
        Handoff["Agent Handoffs"]
        Conflict["Conflict Resolution"]
    end

    CrewDefinition --> TaskManagement
    TaskManagement --> ExecutionModes
    ExecutionModes --> Communication
    Communication --> Results["Aggregated Results"]

    style CrewDefinition fill:#e3f2fd
    style ExecutionModes fill:#e8f5e9
```

### Workflow Engine

```mermaid
flowchart TB
    subgraph WorkflowDef["Workflow Definition"]
        StateGraph["State Graph"]
        Nodes["Workflow Nodes"]
        Edges["Conditional Edges"]
        Transitions["State Transitions"]
    end

    subgraph Executor["Workflow Executor"]
        CurrentState["Current State"]
        EventStream["Event Stream"]
        ErrorHandler["Error Handler"]
    end

    subgraph Routing["Routing Logic"]
        Conditions["Condition Evaluation"]
        NextState["Next State Selection"]
        Fallback["Fallback Paths"]
    end

    WorkflowDef --> Executor
    Executor --> Routing
    Routing --> |"State Change"| Executor
    Executor --> Output["Workflow Output"]

    style WorkflowDef fill:#fff3e0
```

### RAG Pipeline

```mermaid
flowchart LR
    subgraph Ingestion["Document Ingestion"]
        Documents["Source Documents"]
        Chunker["Document Chunker"]
        Embedder["Embedding Generator"]
    end

    subgraph Storage["Vector Storage"]
        VectorStore["In-Memory Vector Store"]
        Index["Similarity Index"]
    end

    subgraph Retrieval["Retrieval"]
        Query["User Query"]
        Search["Semantic Search"]
        TopK["Top-K Results"]
    end

    subgraph Generation["Augmented Generation"]
        Context["Context Assembly"]
        Prompt["Prompt Construction"]
        LLM["LLM Generation"]
    end

    Documents --> Chunker --> Embedder --> VectorStore
    Query --> Search --> VectorStore
    VectorStore --> TopK --> Context
    Context --> Prompt --> LLM --> Response["Generated Response"]
```

### Delegation System

```mermaid
flowchart TB
    IncomingTask["Incoming Task"]

    subgraph Routing["Task Routing"]
        Rules["Routing Rules"]
        SkillMatch["Skill Matching"]
        LoadBalance["Load Balancing"]
        Complexity["Complexity Analysis"]
    end

    subgraph Agents["Agent Pool"]
        Specialist1["Specialist Agent 1"]
        Specialist2["Specialist Agent 2"]
        Generalist["Generalist Agent"]
        Fallback["Fallback Agent"]
    end

    subgraph Fallbacks["Fallback Strategies"]
        Escalate["Escalate to Human"]
        Retry["Retry with Different Agent"]
        Decompose["Decompose Task"]
    end

    IncomingTask --> Routing
    Routing --> |"Best Match"| Agents
    Agents --> |"Success"| Result["Task Result"]
    Agents --> |"Failure"| Fallbacks
    Fallbacks --> |"Retry"| Routing
    Fallbacks --> |"Human Review"| HITL["Human-in-the-Loop"]
```

---

## Data Flow Patterns

### Pattern 1: Linear Phase Flow

```mermaid
flowchart LR
    Input["Phase Input"]

    subgraph Phase["Phase Execution"]
        Process["Processing"]
        Output["Phase Output"]
        Events["Event Emission"]
    end

    subgraph QG["Quality Gate"]
        Assess["Assessment"]
        Decision{"Pass?"}
    end

    Input --> Phase
    Phase --> QG
    Decision --> |"Yes"| NextPhase["Next Phase"]
    Decision --> |"No"| Remediate["Remediation"]
    Remediate --> Phase
```

### Pattern 2: Parallel Production

```mermaid
flowchart TB
    Spec["Asset Specification"]

    subgraph Parallel["Parallel Execution"]
        direction LR
        Track1["Track 1: Models"]
        Track2["Track 2: Textures"]
        Track3["Track 3: Animation"]
        Track4["Track 4: Audio"]
    end

    Spec --> Track1
    Spec --> Track2
    Spec --> Track3
    Spec --> Track4

    Track1 --> Merge["Merge Results"]
    Track2 --> Merge
    Track3 --> Merge
    Track4 --> Merge

    Merge --> Validate["Validation"]
    Validate --> Next["Next Phase"]
```

### Pattern 3: Event-Driven Queue

```mermaid
flowchart TB
    subgraph Producer["Job Producer"]
        API["API Endpoint"]
        Scheduler["Scheduler"]
        Trigger["Event Trigger"]
    end

    subgraph Queue["Job Queue"]
        Pending["Pending Jobs"]
        Active["Active Jobs"]
        Completed["Completed Jobs"]
        Failed["Failed Jobs"]
    end

    subgraph Consumer["Job Consumer"]
        Worker["Worker Process"]
        Handler["Job Handler"]
        Result["Result Processor"]
    end

    subgraph Notifications["Notifications"]
        WebSocket["WebSocket"]
        Webhook["Webhook"]
        Email["Email"]
    end

    Producer --> Queue
    Queue --> Consumer
    Consumer --> Notifications
```

### Pattern 4: Agent Collaboration

```mermaid
flowchart TB
    Task["Complex Task"]

    subgraph Decomposition["Task Decomposition"]
        Analyze["Analyze Task"]
        Split["Split into Subtasks"]
        Assign["Assign to Agents"]
    end

    subgraph Execution["Agent Execution"]
        Agent1["Agent 1"]
        Agent2["Agent 2"]
        Agent3["Agent 3"]

        subgraph Communication["Inter-Agent Comm"]
            Handoff["Task Handoff"]
            Share["Knowledge Sharing"]
            Resolve["Conflict Resolution"]
        end
    end

    subgraph Aggregation["Result Aggregation"]
        Collect["Collect Results"]
        Merge["Merge Outputs"]
        Validate["Validate Consistency"]
    end

    Task --> Decomposition
    Decomposition --> Execution
    Execution --> Aggregation
    Aggregation --> Final["Final Result"]
```

### Pattern 5: Checkpoint/Recovery

```mermaid
flowchart TB
    subgraph Execution["Pipeline Execution"]
        Phase1["Phase 1"] --> CP1["Checkpoint 1"]
        CP1 --> Phase2["Phase 2"]
        Phase2 --> CP2["Checkpoint 2"]
        CP2 --> Phase3["Phase 3"]
        Phase3 --> CP3["Checkpoint 3"]
    end

    subgraph Storage["Checkpoint Storage"]
        State["Execution State"]
        Artifacts["Generated Artifacts"]
        Context["Context Data"]
    end

    subgraph Recovery["Recovery Flow"]
        Failure["Failure Detected"]
        LoadCP["Load Checkpoint"]
        Restore["Restore State"]
        Resume["Resume Execution"]
    end

    CP1 --> Storage
    CP2 --> Storage
    CP3 --> Storage

    Failure --> LoadCP
    Storage --> LoadCP
    LoadCP --> Restore --> Resume
    Resume --> |"From Phase 2"| Phase2

    style CP1 fill:#c8e6c9
    style CP2 fill:#c8e6c9
    style CP3 fill:#c8e6c9
```

---

## Component Interaction Summary

### Cross-System Communication

```mermaid
flowchart TB
    subgraph Systems["Core Systems"]
        direction LR
        Pipelines["Pipelines"]
        Agents["Agents"]
        Workers["Workers"]
        QualityGates["Quality Gates"]
    end

    subgraph Communication["Communication Channels"]
        Events["TypedEventEmitter"]
        Queues["BullMQ Queues"]
        GRPC["gRPC Streaming"]
        WS["WebSocket"]
        Webhooks["Webhooks"]
    end

    subgraph State["State Management"]
        DB[(PostgreSQL)]
        Cache[(Redis)]
        Storage[(MinIO)]
    end

    Systems --> Communication
    Communication --> State

    Events --> |"Internal Events"| Systems
    Queues --> |"Async Jobs"| Systems
    GRPC --> |"Service Calls"| Systems
    WS --> |"Real-time Updates"| Clients["Clients"]
    Webhooks --> |"External Notifications"| External["External Systems"]
```

---

## Key Files Reference

| Component         | Location                                             | Lines   |
| ----------------- | ---------------------------------------------------- | ------- |
| Game Pipeline     | `packages/agents/src/pipelines/game-pipeline.ts`     | 3,336   |
| Film Pipeline     | `packages/agents/src/pipelines/film-pipeline.ts`     | 2,345   |
| Infrastructure    | `packages/agents/src/pipelines/infrastructure.ts`    | 1,716   |
| Quality Gates     | `packages/agents/src/pipelines/quality-gates.ts`     | 3,033   |
| Automated Testing | `packages/agents/src/pipelines/automated-testing.ts` | 2,154   |
| Crew System       | `packages/agents/src/orchestration/crew.ts`          | ~500    |
| Workflow Engine   | `packages/agents/src/orchestration/workflow.ts`      | ~400    |
| Workers           | `apps/workers/src/workers/`                          | Various |
| 3D Generation     | `packages/3d-generation/src/`                        | Various |

---

## See Also

- [API Endpoints Reference](../api/endpoints/README.md)
- [Deployment Guide](./deployment.md)
- [Troubleshooting Guide](./troubleshooting.md)
