Eve library

Pipeline Architecture

Guide

Revision 1

On this page

Comprehensive architecture diagrams for all Yemaya platform pipelines.

Table of Contents#

  1. System Overview
  2. Game Creation Pipeline
  3. Film Creation Pipeline
  4. Pipeline Infrastructure
  5. Quality Gates System
  6. Worker Queue System
  7. 3D Generation Pipeline
  8. Agent Orchestration System
  9. Data Flow Patterns

System Overview#

High-Level Architecture#

flowchart TB subgraph Clients["Client Applications"] Web["Web App"] Desktop["Desktop App"] CLI["CLI Tool"] SDK["TypeScript/Python SDK"] end subgraph API["API Layer"] REST["REST API<br/>(Hono)"] WS["WebSocket<br/>(Real-time)"] GRPC["gRPC Services<br/>(Protobuf)"] end subgraph Pipelines["Pipeline Orchestration"] GamePipe["Game Creation<br/>Pipeline"] FilmPipe["Film Creation<br/>Pipeline"] ThreeD["3D Generation<br/>Pipeline"] Audio["Audio Pipeline"] end subgraph Orchestration["Agent Orchestration"] Crew["Crew System"] Workflow["Workflow Engine"] Delegation["Task Delegation"] RAG["RAG Pipeline"] end subgraph Workers["Worker System (BullMQ)"] AgentWorker["Agent Worker"] MediaWorker["Media Worker"] AssetWorker["Asset Worker"] Gen3DWorker["3D Gen Worker"] NotifyWorker["Notification Worker"] end subgraph QualityGates["Quality Gates"] Narrative["Narrative QG"] Visual["Visual QG"] AudioQG["Audio QG"] Technical["Technical QG"] Gameplay["Gameplay QG"] end subgraph Infrastructure["Infrastructure"] Postgres[(PostgreSQL)] Redis[(Redis)] MinIO[(MinIO/S3)] CDN["CDN"] end subgraph External["External Services"] AI["AI Providers<br/>(Claude, OpenAI, Gemini)"] ThreeDProviders["3D Providers<br/>(Rodin, Meshy, etc.)"] GameEngines["Game Engines<br/>(Blender, Godot, Unreal)"] end Clients --> API API --> Pipelines Pipelines --> Orchestration Pipelines --> Workers Pipelines --> QualityGates Workers --> Infrastructure Workers --> External Orchestration --> AI ThreeD --> ThreeDProviders GamePipe --> GameEngines FilmPipe --> GameEngines

Pipeline Types Comparison#

FlowchartFlowchart: 10 steps and 5 connections.Concept → ReleaseDevelopment → DeliveryExecution SupportValidationVerificationPipeline TypesGame Pipeline 8 Phases3,336 LOCFilm Pipeline 5 Phases2,345 LOCInfrastructure 4Components 1,716 LOCQuality Gates 5 Types3,033 LOCAutomated Testing 4Categories 2,154 LOCPlayable GameFinal FilmPipeline OpsQuality AssuranceTest Reports
Flowchart: Flowchart. Flowchart: 10 steps and 5 connections.

Drawn from its Mermaid source.

As text
Groups:
- Pipeline Types: Game Pipeline 8 Phases 3,336 LOC, Film Pipeline 5 Phases 2,345 LOC, Infrastructure 4 Components 1,716 LOC, Quality Gates 5 Types 3,033 LOC, Automated Testing 4 Categories 2,154 LOC

Parts:
- Game Pipeline 8 Phases 3,336 LOC, in Pipeline Types
- Film Pipeline 5 Phases 2,345 LOC, in Pipeline Types
- Infrastructure 4 Components 1,716 LOC, in Pipeline Types
- Quality Gates 5 Types 3,033 LOC, in Pipeline Types
- Automated Testing 4 Categories 2,154 LOC, in Pipeline Types
- Playable Game
- Final Film
- Pipeline Ops
- Quality Assurance
- Test Reports

Connections:
- Game Pipeline 8 Phases 3,336 LOC to Playable Game: Concept → Release
- Film Pipeline 5 Phases 2,345 LOC to Final Film: Development → Delivery
- Infrastructure 4 Components 1,716 LOC to Pipeline Ops: Execution Support
- Quality Gates 5 Types 3,033 LOC to Quality Assurance: Validation
- Automated Testing 4 Categories 2,154 LOC to Test Reports: Verification
Show source
graph LR
    subgraph Types["Pipeline Types"]
        direction TB
        Game["Game Pipeline<br/>8 Phases<br/>3,336 LOC"]
        Film["Film Pipeline<br/>5 Phases<br/>2,345 LOC"]
        Infra["Infrastructure<br/>4 Components<br/>1,716 LOC"]
        QG["Quality Gates<br/>5 Types<br/>3,033 LOC"]
        Test["Automated Testing<br/>4 Categories<br/>2,154 LOC"]
    end

    Game --> |"Concept → Release"| GameOut["Playable Game"]
    Film --> |"Development → Delivery"| FilmOut["Final Film"]
    Infra --> |"Execution Support"| InfraOut["Pipeline Ops"]
    QG --> |"Validation"| QGOut["Quality Assurance"]
    Test --> |"Verification"| TestOut["Test Reports"]

Game Creation Pipeline#

Complete Phase Flow#

flowchart TB subgraph Phase1["Phase 1: Concept Generation"] UserPrompt["User Prompt"] GameConcept["Game Concept"] ArtDirection["Art Direction"] MarketAnalysis["Market Analysis"] MoodBoards["Mood Boards"] end subgraph Phase2["Phase 2: GDD Generation"] CoreLoop["Core Loop Design"] Mechanics["Mechanic Design"] Systems["System Design"] Levels["Level Design"] Narrative["Narrative Design"] end subgraph Phase3["Phase 3: Prototype"] Prototype["Playable Prototype"] Metrics["Prototype Metrics"] Feedback["Playtester Feedback"] Iteration["Prototype Iteration"] end subgraph Phase4["Phase 4: Asset Production"] AssetPlan["Asset Production Plan"] Models["3D Models"] Animations["Animations"] Textures["Textures"] VFX["Visual Effects"] Audio["Audio Assets"] end subgraph Phase5["Phase 5: Level Assembly"] LevelAssembly["Level Construction"] Scripting["Gameplay Scripting"] Triggers["Trigger Setup"] Lighting["Lighting Setup"] UI["UI Implementation"] end subgraph Phase6["Phase 6: Polish & Optimization"] BugFix["Bug Fixing"] PerfOpt["Performance Optimization"] Balance["Balance Tuning"] Juice["Juice Pass"] Accessibility["Accessibility"] end subgraph Phase7["Phase 7: QA & Testing"] TestStrategy["Testing Strategy"] AutoTests["Automated Tests"] AIPlaytest["AI Playtesting"] ManualTest["Manual Testing"] end subgraph Phase8["Phase 8: Release"] PlatformBuild["Platform Builds"] StoreSubmit["Store Submission"] Marketing["Marketing"] Launch["Launch"] end Phase1 --> |QualityGate| Phase2 Phase2 --> |QualityGate| Phase3 Phase3 --> |QualityGate| Phase4 Phase4 --> |QualityGate| Phase5 Phase5 --> |QualityGate| Phase6 Phase6 --> |QualityGate| Phase7 Phase7 --> |QualityGate| Phase8 Phase8 --> Released["Released Game"] style Phase1 fill:#e1f5fe style Phase2 fill:#e8f5e9 style Phase3 fill:#fff3e0 style Phase4 fill:#fce4ec style Phase5 fill:#f3e5f5 style Phase6 fill:#e0f2f1 style Phase7 fill:#fff8e1 style Phase8 fill:#efebe9

Asset Production Parallel Flow#

flowchart LR AssetSpec["Asset Specification<br/>(from GDD)"] subgraph Parallel["Parallel Production"] direction TB subgraph ModelPipe["3D Model Pipeline"] Generate3D["Generate 3D"] MeshProcess["Mesh Processing"] LODGen["LOD Generation"] end subgraph TexturePipe["Texture Pipeline"] GenTexture["Generate Textures"] PBRSetup["PBR Setup"] BakeMaps["Bake Maps"] end subgraph AnimPipe["Animation Pipeline"] Rigging["Auto-Rigging"] AnimGen["Animation Gen"] Retarget["Retargeting"] end subgraph AudioPipe["Audio Pipeline"] SFXGen["SFX Generation"] MusicGen["Music Generation"] VoiceSynth["Voice Synthesis"] end end AssetSpec --> ModelPipe AssetSpec --> TexturePipe AssetSpec --> AnimPipe AssetSpec --> AudioPipe ModelPipe --> Merge["Asset Merge"] TexturePipe --> Merge AnimPipe --> Merge AudioPipe --> Merge Merge --> Validate["Validation"] Validate --> Export["Export to Engine"]

Film Creation Pipeline#

Complete Phase Flow#

flowchart TB subgraph Dev["Phase 1: Development"] Concept["Film Concept"] Treatment["Treatment"] PitchDeck["Pitch Deck"] Budget["Budget Estimate"] Casting["Casting Suggestions"] Locations["Location Suggestions"] end subgraph PreProd["Phase 2: Pre-Production"] Screenplay["Screenplay"] Storyboard["Storyboards"] ShotList["Shot Lists"] Schedule["Production Schedule"] BudgetBreak["Budget Breakdown"] CallSheets["Call Sheets"] end subgraph Prod["Phase 3: Animation Production"] Environments["Environment Assets"] Characters["Character Assets"] Props["Prop Assets"] Animation["Animation"] LightingSetup["Lighting Setup"] Rendering["Rendering"] end subgraph Post["Phase 4: Post-Production"] EditAssembly["Edit Assembly"] VFX["VFX Integration"] ColorGrade["Color Grading"] SoundDesign["Sound Design"] MusicScore["Music Scoring"] FinalMix["Final Mix"] end subgraph Delivery["Phase 5: Delivery"] DCP["DCP Creation"] Transcode["Transcoding"] QC["Quality Control"] Distribute["Distribution"] Subtitles["Subtitles"] Dubbing["Dubbing"] end Dev --> |Greenlight| PreProd PreProd --> |Approved| Prod Prod --> |Rendered| Post Post --> |Final| Delivery Delivery --> Released["Released Film"] style Dev fill:#e3f2fd style PreProd fill:#e8f5e9 style Prod fill:#fff3e0 style Post fill:#fce4ec style Delivery fill:#f3e5f5

Post-Production Parallel Workflow#

flowchart TB RenderedShots["Rendered Shots"] subgraph PostParallel["Parallel Post-Production"] direction LR subgraph EditTrack["Edit Track"] RoughCut["Rough Cut"] FineCut["Fine Cut"] FinalCut["Final Cut"] end subgraph VFXTrack["VFX Track"] VFXPlanning["VFX Planning"] VFXExec["VFX Execution"] VFXComp["Compositing"] end subgraph AudioTrack["Audio Track"] Dialogue["Dialogue Edit"] Foley["Foley"] Ambience["Ambience"] end subgraph MusicTrack["Music Track"] Temp["Temp Score"] Compose["Composition"] Record["Recording"] end end RenderedShots --> PostParallel PostParallel --> ColorGrade["Color Grading"] ColorGrade --> FinalMix["Final Mix"] FinalMix --> Master["Master Delivery"]

Pipeline Infrastructure#

Executor Architecture#

flowchart TB subgraph Executor["PipelineExecutor"] PhaseQueue["Phase Queue"] subgraph Execution["Execution Engine"] StateManager["State Manager"] CheckpointMgr["Checkpoint Manager"] RetryPolicy["Retry Policy"] RollbackMgr["Rollback Manager"] end subgraph ResourceMgmt["Resource Management"] Scheduler["Resource Scheduler"] Allocator["Resource Allocator"] CostTracker["Cost Tracker"] Utilization["Utilization Monitor"] end subgraph Monitoring["Monitoring"] Monitor["Pipeline Monitor"] Alerts["Alert System"] Dashboard["Dashboard"] Analytics["Analytics"] end end Input["Phase Definition"] --> PhaseQueue PhaseQueue --> Execution Execution --> ResourceMgmt Execution --> Monitoring Execution --> |Success| Output["Phase Output"] Execution --> |Failure| Recovery["Recovery/Rollback"] style Executor fill:#e3f2fd

Retry Policy Strategies#

flowchart LR subgraph Strategies["Retry Strategies"] direction TB Linear["Linear Backoff<br/>delay × attempt"] Exponential["Exponential Backoff<br/>delay × 2^attempt"] Fixed["Fixed Delay<br/>constant delay"] end Failure["Phase Failure"] --> Check{"Attempts<br/>Remaining?"} Check --> |Yes| Strategies Check --> |No| Final["Final Failure"] Strategies --> Wait["Wait Period"] Wait --> Retry["Retry Execution"] Retry --> |Success| Continue["Continue Pipeline"] Retry --> |Failure| Check

Template System#

flowchart TB subgraph Templates["Pipeline Templates"] direction LR GameTemplate["Game Template"] FilmTemplate["Film Template"] CustomTemplate["Custom Template"] end subgraph Variables["Template Variables"] ProjectName["Project Name"] Genre["Genre"] Platform["Target Platform"] Quality["Quality Level"] end subgraph Overrides["Phase Overrides"] SkipPhase["Skip Phases"] CustomPhase["Custom Phases"] ModifyPhase["Modified Phases"] end Templates --> Instantiate["Template Instantiation"] Variables --> Instantiate Overrides --> Instantiate Instantiate --> Pipeline["Configured Pipeline"] Pipeline --> Execute["Execute"]

Quality Gates System#

Quality Gate Flow#

flowchart TB PhaseOutput["Phase Output"] subgraph Assessment["Quality Assessment"] Collect["Collect Artifacts"] Analyze["Run Analysis"] Score["Calculate Scores"] end PhaseOutput --> Assessment Assessment --> Decision{"Quality<br/>Decision"} Decision --> |"Score >= Threshold"| Pass["PASS"] Decision --> |"Score < Threshold"| Fail["FAIL"] Decision --> |"Manual Review"| Override["OVERRIDE"] Pass --> Continue["Continue Pipeline"] Fail --> Remediate["Remediation Phase"] Remediate --> |"Fixes Applied"| Assessment Remediate --> |"Cannot Fix"| Block["Block Pipeline"] Override --> |"With Justification"| Continue Override --> |"Rejected"| Block style Pass fill:#c8e6c9 style Fail fill:#ffcdd2 style Override fill:#fff9c4

Quality Gate Types#

flowchart LR subgraph NarrativeGate["Narrative Quality Gate"] direction TB ScriptQuality["Script Quality"] DialogueCheck["Dialogue Check"] CharConsist["Character Consistency"] PlotCoherence["Plot Coherence"] end subgraph VisualGate["Visual Quality Gate"] direction TB StyleConsist["Style Consistency"] TechSpecs["Technical Specs"] ColorPalette["Color Palette"] Optimization["Optimization"] end subgraph AudioGate["Audio Quality Gate"] direction TB LevelCheck["Level Check"] FreqResponse["Frequency Response"] SyncCheck["Sync Verification"] FormatCheck["Format Compliance"] end subgraph TechnicalGate["Technical Quality Gate"] direction TB Performance["Performance"] Stability["Stability"] Compatibility["Compatibility"] Memory["Memory Usage"] end subgraph GameplayGate["Gameplay Quality Gate"] direction TB Balance["Balance"] Difficulty["Difficulty Curve"] FunFactor["Fun Factor"] Pacing["Pacing"] end Input["Phase Output"] --> NarrativeGate Input --> VisualGate Input --> AudioGate Input --> TechnicalGate Input --> GameplayGate NarrativeGate --> Aggregate["Aggregate Score"] VisualGate --> Aggregate AudioGate --> Aggregate TechnicalGate --> Aggregate GameplayGate --> Aggregate Aggregate --> Final["Quality Decision"]

Issue Classification#

flowchart TB subgraph IssueTypes["Issue Types"] direction LR subgraph Critical["Critical"] Crash["Crashes"] DataLoss["Data Loss"] Security["Security Issues"] end subgraph Major["Major"] Broken["Broken Features"] PerfIssue["Performance Issues"] Regression["Regressions"] end subgraph Minor["Minor"] Visual["Visual Glitches"] Polish["Polish Issues"] Edge["Edge Cases"] end end Critical --> |"Must Fix"| Block["Block Release"] Major --> |"Should Fix"| Remediate["Remediation Required"] Minor --> |"Can Ship"| Track["Track for Future"] style Critical fill:#ffcdd2 style Major fill:#fff9c4 style Minor fill:#c8e6c9

Worker Queue System#

BullMQ Architecture#

flowchart TB subgraph API["API Layer"] REST["REST Endpoints"] GRPC["gRPC Services"] WS["WebSocket"] end subgraph Queues["BullMQ Queues"] AgentQ["yemaya:agent"] MediaQ["yemaya:media"] AssetQ["yemaya:asset"] Gen3DQ["yemaya:generation-3d"] NotifyQ["yemaya:notification"] end subgraph Workers["Worker Processes"] AgentW["Agent Worker<br/>3 retries, exponential"] MediaW["Media Worker<br/>2 retries, fixed"] AssetW["Asset Worker<br/>3 retries"] Gen3DW["3D Gen Worker<br/>3 retries, 7-day retention"] NotifyW["Notification Worker<br/>5 retries"] end subgraph Backend["Queue Backend"] Redis[(Redis)] end API --> |"Add Job"| Queues Queues --> Redis Redis --> Workers Workers --> |"Job Complete"| Events["Event Emission"] Workers --> |"Progress"| WS Events --> |"Webhook"| External["External Systems"] style Redis fill:#ffcdd2

Job Type Details#

FlowchartAn overview: 4 groups. Open a group to see inside it.3D Generation JobsAsset JobsMedia JobsAgent Jobs
Flowchart: Flowchart. Flowchart: 19 steps and 0 connections.

This is an overview. Open a group to see inside it.

Drawn from its Mermaid source.

As text
Groups:
- 3D Generation Jobs: generation-3d.text-to-3d, generation-3d.image-to-3d, generation-3d.mesh-process, generation-3d.auto-rig, generation-3d.texture-generate
- Asset Jobs: asset.optimize-image, asset.generate-variants, asset.extract-metadata, asset.validate
- Media Jobs: media.transcode-video, media.transcode-audio, media.extract-frames, media.generate-thumbnail, media.concatenate
- Agent Jobs: agent.execute-task, agent.batch-process, agent.generate-script, agent.analyze-content, agent.collaborate

Parts:
- agent.execute-task, in Agent Jobs
- agent.batch-process, in Agent Jobs
- agent.generate-script, in Agent Jobs
- agent.analyze-content, in Agent Jobs
- agent.collaborate, in Agent Jobs
- media.transcode-video, in Media Jobs
- media.transcode-audio, in Media Jobs
- media.extract-frames, in Media Jobs
- media.generate-thumbnail, in Media Jobs
- media.concatenate, in Media Jobs
- asset.optimize-image, in Asset Jobs
- asset.generate-variants, in Asset Jobs
- asset.extract-metadata, in Asset Jobs
- asset.validate, in Asset Jobs
- generation-3d.text-to-3d, in 3D Generation Jobs
- generation-3d.image-to-3d, in 3D Generation Jobs
- generation-3d.mesh-process, in 3D Generation Jobs
- generation-3d.auto-rig, in 3D Generation Jobs
- generation-3d.texture-generate, in 3D Generation Jobs
Show source
flowchart LR
    subgraph AgentJobs["Agent Jobs"]
        direction TB
        ExecuteTask["agent.execute-task"]
        BatchProcess["agent.batch-process"]
        GenScript["agent.generate-script"]
        AnalyzeContent["agent.analyze-content"]
        Collaborate["agent.collaborate"]
    end

    subgraph MediaJobs["Media Jobs"]
        direction TB
        TranscodeV["media.transcode-video"]
        TranscodeA["media.transcode-audio"]
        ExtractFrames["media.extract-frames"]
        GenThumb["media.generate-thumbnail"]
        Concatenate["media.concatenate"]
    end

    subgraph AssetJobs["Asset Jobs"]
        direction TB
        OptimizeImg["asset.optimize-image"]
        GenVariants["asset.generate-variants"]
        ExtractMeta["asset.extract-metadata"]
        Validate["asset.validate"]
    end

    subgraph Gen3DJobs["3D Generation Jobs"]
        direction TB
        TextTo3D["generation-3d.text-to-3d"]
        ImageTo3D["generation-3d.image-to-3d"]
        MeshProcess["generation-3d.mesh-process"]
        AutoRig["generation-3d.auto-rig"]
        TextureGen["generation-3d.texture-generate"]
    end

Job Lifecycle#

State diagramState diagram: 7 states and 9 transitions.Job AddedWorker ClaimsSuccessErrorRetryMax RetriesDoneRate LimitedDelay ExpiresWaitingActiveCompletedFailedDelayed
State machine: State diagram. State diagram: 7 states and 9 transitions.

Drawn from its Mermaid source.

As text
Parts:
- Start (initial)
- Waiting
- Active
- Completed
- Failed
- End (final)
- Delayed

Connections:
- Start to Waiting: Job Added
- Waiting to Active: Worker Claims
- Active to Completed: Success
- Active to Failed: Error
- Failed to Waiting: Retry
- Failed to End: Max Retries
- Completed to End: Done
- Active to Delayed: Rate Limited
- Delayed to Waiting: Delay Expires
Show source
stateDiagram-v2
    [*] --> Waiting: Job Added
    Waiting --> Active: Worker Claims
    Active --> Completed: Success
    Active --> Failed: Error
    Failed --> Waiting: Retry
    Failed --> [*]: Max Retries
    Completed --> [*]: Done

    Active --> Delayed: Rate Limited
    Delayed --> Waiting: Delay Expires

3D Generation Pipeline#

Provider Architecture#

flowchart TB subgraph Input["Input Types"] TextPrompt["Text Prompt"] Image["Reference Image"] MultiView["Multi-View Images"] end subgraph Providers["3D Generation Providers"] Rodin["Rodin"] Meshy["Meshy"] Tripo["Tripo"] Trellis["Trellis"] Hunyuan["Hunyuan"] Threedfy["Threedfy"] Local["Local Model"] end subgraph Processing["Mesh Processing"] Optimize["Optimization"] Cleanup["Mesh Cleanup"] UVUnwrap["UV Unwrapping"] LODGen["LOD Generation"] end subgraph Texturing["Texture Processing"] PBRGen["PBR Generation"] TextureApply["Texture Application"] BakeMaps["Map Baking"] end subgraph Export["Export Formats"] GLB["GLB"] GLTF["GLTF"] FBX["FBX"] OBJ["OBJ"] USD["USD/USDZ"] end Input --> |"Provider Selection"| Providers Providers --> |"Raw Mesh"| Processing Processing --> |"Clean Mesh"| Texturing Texturing --> |"Textured Model"| Export style Providers fill:#e3f2fd style Processing fill:#e8f5e9 style Texturing fill:#fff3e0

Quality Levels#

flowchart LR subgraph QualityLevels["Quality Levels"] Draft["Draft<br/>Low poly, fast"] Standard["Standard<br/>Balanced"] High["High<br/>Detailed"] Ultra["Ultra<br/>Maximum quality"] end Draft --> |"~30s"| DraftOut["Preview Model"] Standard --> |"~2min"| StdOut["Production Ready"] High --> |"~5min"| HighOut["High Detail"] Ultra --> |"~10min"| UltraOut["Hero Asset"] style Draft fill:#fff9c4 style Standard fill:#c8e6c9 style High fill:#bbdefb style Ultra fill:#e1bee7

Complete Generation Flow#

sequenceDiagram participant Client participant API participant Queue participant Worker participant Provider participant Storage Client->>API: POST /3d/generate API->>Queue: Add Job API-->>Client: { jobId, status: "queued" } Queue->>Worker: Process Job Worker->>Provider: Generate 3D Model loop Progress Updates Provider-->>Worker: Progress % Worker-->>API: WebSocket Update API-->>Client: Progress Event end Provider-->>Worker: Raw Model Worker->>Worker: Mesh Processing Worker->>Worker: Texture Processing Worker->>Storage: Upload Assets Worker->>API: Job Complete API-->>Client: Completion Event Client->>API: GET /3d/jobs/{id} API-->>Client: { status: "completed", assets: [...] }

Agent Orchestration System#

Crew System Architecture#

flowchart TB subgraph CrewDefinition["Crew Definition"] Roles["Agent Roles"] Goals["Role Goals"] Tools["Available Tools"] Backstory["Agent Backstory"] end subgraph TaskManagement["Task Management"] Tasks["Task Queue"] Dependencies["Task Dependencies"] Priority["Priority Assignment"] end subgraph ExecutionModes["Execution Modes"] Sequential["Sequential<br/>One-by-one"] Parallel["Parallel<br/>Concurrent"] Hierarchical["Hierarchical<br/>Manager delegates"] end subgraph Communication["Communication"] Bus["Crew Message Bus"] Handoff["Agent Handoffs"] Conflict["Conflict Resolution"] end CrewDefinition --> TaskManagement TaskManagement --> ExecutionModes ExecutionModes --> Communication Communication --> Results["Aggregated Results"] style CrewDefinition fill:#e3f2fd style ExecutionModes fill:#e8f5e9

Workflow Engine#

flowchart TB subgraph WorkflowDef["Workflow Definition"] StateGraph["State Graph"] Nodes["Workflow Nodes"] Edges["Conditional Edges"] Transitions["State Transitions"] end subgraph Executor["Workflow Executor"] CurrentState["Current State"] EventStream["Event Stream"] ErrorHandler["Error Handler"] end subgraph Routing["Routing Logic"] Conditions["Condition Evaluation"] NextState["Next State Selection"] Fallback["Fallback Paths"] end WorkflowDef --> Executor Executor --> Routing Routing --> |"State Change"| Executor Executor --> Output["Workflow Output"] style WorkflowDef fill:#fff3e0

RAG Pipeline#

FlowchartFlowchart: 12 steps and 10 connections.Augmented GenerationRetrievalVector StorageDocument IngestionSource DocumentsDocument ChunkerEmbedding GeneratorIn-Memory Vector StoreSimilarity IndexUser QuerySemantic SearchTop-K ResultsContext AssemblyPrompt ConstructionLLM GenerationGenerated Response
Flowchart: Flowchart. Flowchart: 12 steps and 10 connections.

Drawn from its Mermaid source.

As text
Groups:
- Augmented Generation: Context Assembly, Prompt Construction, LLM Generation
- Retrieval: User Query, Semantic Search, Top-K Results
- Vector Storage: In-Memory Vector Store, Similarity Index
- Document Ingestion: Source Documents, Document Chunker, Embedding Generator

Parts:
- Source Documents, in Document Ingestion
- Document Chunker, in Document Ingestion
- Embedding Generator, in Document Ingestion
- In-Memory Vector Store, in Vector Storage
- Similarity Index, in Vector Storage
- User Query, in Retrieval
- Semantic Search, in Retrieval
- Top-K Results, in Retrieval
- Context Assembly, in Augmented Generation
- Prompt Construction, in Augmented Generation
- LLM Generation, in Augmented Generation
- Generated Response

Connections:
- Source Documents to Document Chunker
- Document Chunker to Embedding Generator
- Embedding Generator to In-Memory Vector Store
- User Query to Semantic Search
- Semantic Search to In-Memory Vector Store
- In-Memory Vector Store to Top-K Results
- Top-K Results to Context Assembly
- Context Assembly to Prompt Construction
- Prompt Construction to LLM Generation
- LLM Generation to Generated Response
Show source
flowchart LR
    subgraph Ingestion["Document Ingestion"]
        Documents["Source Documents"]
        Chunker["Document Chunker"]
        Embedder["Embedding Generator"]
    end

    subgraph Storage["Vector Storage"]
        VectorStore["In-Memory Vector Store"]
        Index["Similarity Index"]
    end

    subgraph Retrieval["Retrieval"]
        Query["User Query"]
        Search["Semantic Search"]
        TopK["Top-K Results"]
    end

    subgraph Generation["Augmented Generation"]
        Context["Context Assembly"]
        Prompt["Prompt Construction"]
        LLM["LLM Generation"]
    end

    Documents --> Chunker --> Embedder --> VectorStore
    Query --> Search --> VectorStore
    VectorStore --> TopK --> Context
    Context --> Prompt --> LLM --> Response["Generated Response"]

Delegation System#

flowchart TB IncomingTask["Incoming Task"] subgraph Routing["Task Routing"] Rules["Routing Rules"] SkillMatch["Skill Matching"] LoadBalance["Load Balancing"] Complexity["Complexity Analysis"] end subgraph Agents["Agent Pool"] Specialist1["Specialist Agent 1"] Specialist2["Specialist Agent 2"] Generalist["Generalist Agent"] Fallback["Fallback Agent"] end subgraph Fallbacks["Fallback Strategies"] Escalate["Escalate to Human"] Retry["Retry with Different Agent"] Decompose["Decompose Task"] end IncomingTask --> Routing Routing --> |"Best Match"| Agents Agents --> |"Success"| Result["Task Result"] Agents --> |"Failure"| Fallbacks Fallbacks --> |"Retry"| Routing Fallbacks --> |"Human Review"| HITL["Human-in-the-Loop"]

Data Flow Patterns#

Pattern 1: Linear Phase Flow#

flowchart LR Input["Phase Input"] subgraph Phase["Phase Execution"] Process["Processing"] Output["Phase Output"] Events["Event Emission"] end subgraph QG["Quality Gate"] Assess["Assessment"] Decision{"Pass?"} end Input --> Phase Phase --> QG Decision --> |"Yes"| NextPhase["Next Phase"] Decision --> |"No"| Remediate["Remediation"] Remediate --> Phase

Pattern 2: Parallel Production#

FlowchartFlowchart: 8 steps and 10 connections.Parallel ExecutionAsset SpecificationTrack 1: ModelsTrack 2: TexturesTrack 3: AnimationTrack 4: AudioMerge ResultsValidationNext Phase
Flowchart: Flowchart. Flowchart: 8 steps and 10 connections.

Drawn from its Mermaid source.

As text
Groups:
- Parallel Execution: Track 1: Models, Track 2: Textures, Track 3: Animation, Track 4: Audio

Parts:
- Asset Specification
- Track 1: Models, in Parallel Execution
- Track 2: Textures, in Parallel Execution
- Track 3: Animation, in Parallel Execution
- Track 4: Audio, in Parallel Execution
- Merge Results
- Validation
- Next Phase

Connections:
- Asset Specification to Track 1: Models
- Asset Specification to Track 2: Textures
- Asset Specification to Track 3: Animation
- Asset Specification to Track 4: Audio
- Track 1: Models to Merge Results
- Track 2: Textures to Merge Results
- Track 3: Animation to Merge Results
- Track 4: Audio to Merge Results
- Merge Results to Validation
- Validation to Next Phase
Show source
flowchart TB
    Spec["Asset Specification"]

    subgraph Parallel["Parallel Execution"]
        direction LR
        Track1["Track 1: Models"]
        Track2["Track 2: Textures"]
        Track3["Track 3: Animation"]
        Track4["Track 4: Audio"]
    end

    Spec --> Track1
    Spec --> Track2
    Spec --> Track3
    Spec --> Track4

    Track1 --> Merge["Merge Results"]
    Track2 --> Merge
    Track3 --> Merge
    Track4 --> Merge

    Merge --> Validate["Validation"]
    Validate --> Next["Next Phase"]

Pattern 3: Event-Driven Queue#

flowchart TB subgraph Producer["Job Producer"] API["API Endpoint"] Scheduler["Scheduler"] Trigger["Event Trigger"] end subgraph Queue["Job Queue"] Pending["Pending Jobs"] Active["Active Jobs"] Completed["Completed Jobs"] Failed["Failed Jobs"] end subgraph Consumer["Job Consumer"] Worker["Worker Process"] Handler["Job Handler"] Result["Result Processor"] end subgraph Notifications["Notifications"] WebSocket["WebSocket"] Webhook["Webhook"] Email["Email"] end Producer --> Queue Queue --> Consumer Consumer --> Notifications

Pattern 4: Agent Collaboration#

flowchart TB Task["Complex Task"] subgraph Decomposition["Task Decomposition"] Analyze["Analyze Task"] Split["Split into Subtasks"] Assign["Assign to Agents"] end subgraph Execution["Agent Execution"] Agent1["Agent 1"] Agent2["Agent 2"] Agent3["Agent 3"] subgraph Communication["Inter-Agent Comm"] Handoff["Task Handoff"] Share["Knowledge Sharing"] Resolve["Conflict Resolution"] end end subgraph Aggregation["Result Aggregation"] Collect["Collect Results"] Merge["Merge Outputs"] Validate["Validate Consistency"] end Task --> Decomposition Decomposition --> Execution Execution --> Aggregation Aggregation --> Final["Final Result"]

Pattern 5: Checkpoint/Recovery#

flowchart TB subgraph Execution["Pipeline Execution"] Phase1["Phase 1"] --> CP1["Checkpoint 1"] CP1 --> Phase2["Phase 2"] Phase2 --> CP2["Checkpoint 2"] CP2 --> Phase3["Phase 3"] Phase3 --> CP3["Checkpoint 3"] end subgraph Storage["Checkpoint Storage"] State["Execution State"] Artifacts["Generated Artifacts"] Context["Context Data"] end subgraph Recovery["Recovery Flow"] Failure["Failure Detected"] LoadCP["Load Checkpoint"] Restore["Restore State"] Resume["Resume Execution"] end CP1 --> Storage CP2 --> Storage CP3 --> Storage Failure --> LoadCP Storage --> LoadCP LoadCP --> Restore --> Resume Resume --> |"From Phase 2"| Phase2 style CP1 fill:#c8e6c9 style CP2 fill:#c8e6c9 style CP3 fill:#c8e6c9

Component Interaction Summary#

Cross-System Communication#

flowchart TB subgraph Systems["Core Systems"] direction LR Pipelines["Pipelines"] Agents["Agents"] Workers["Workers"] QualityGates["Quality Gates"] end subgraph Communication["Communication Channels"] Events["TypedEventEmitter"] Queues["BullMQ Queues"] GRPC["gRPC Streaming"] WS["WebSocket"] Webhooks["Webhooks"] end subgraph State["State Management"] DB[(PostgreSQL)] Cache[(Redis)] Storage[(MinIO)] end Systems --> Communication Communication --> State Events --> |"Internal Events"| Systems Queues --> |"Async Jobs"| Systems GRPC --> |"Service Calls"| Systems WS --> |"Real-time Updates"| Clients["Clients"] Webhooks --> |"External Notifications"| External["External Systems"]

Key Files Reference#

Component Location Lines
Game Pipeline packages/agents/src/pipelines/game-pipeline.ts 3,336
Film Pipeline packages/agents/src/pipelines/film-pipeline.ts 2,345
Infrastructure packages/agents/src/pipelines/infrastructure.ts 1,716
Quality Gates packages/agents/src/pipelines/quality-gates.ts 3,033
Automated Testing packages/agents/src/pipelines/automated-testing.ts 2,154
Crew System packages/agents/src/orchestration/crew.ts ~500
Workflow Engine packages/agents/src/orchestration/workflow.ts ~400
Workers apps/workers/src/workers/ Various
3D Generation packages/3d-generation/src/ Various

See Also#