# Living World & Autonomous NPC SOTA — Beyond GTA 6 ## Executive Summary This proposal identifies **7 critical system gaps** between the Oshun/Maya engine ecosystem's existing plans and what would be required to build the most advanced living world and autonomous NPC system ever created — surpassing GTA 6 (Rockstar), Cyberpunk 2077 (CDPR), RDR2, The Sims, Dwarf Fortress, and everything publicly known from NVIDIA ACE, Google SIMA 2, and Stanford Generative Agents. The existing TODOS already cover substantial ground: - **Phase 40.10** — maya/souls LLM NPC foundations (all marked done) - **Phase 43** — Neith AI Runtime: navmesh, behavior trees, state machines (done) - **Phase 72** — Neural rendering, game engine SOTA gap closure - **Phase 78** — Voice ASR pipeline for modding companion - **Phase 79.11** — Crowd simulation at 10K+ NPCs - **Phase 79.24-36** — NPC life simulation, relationships, occupations, cities, economy, law, politics, gossip, animals, culture, history - **Phase 80** — Social fabric for MMO community This proposal fills the **unfilled gaps** — the systems that would make Oshun's NPCs not just technically competitive but categorically different from anything that exists. --- ## What This Proposal Covers (The 7 Gaps) | Gap | System | Why It Doesn't Exist Yet | |-----|--------|--------------------------| | **1** | On-Device SLM Cognitive Core | Phase 40.10 is cloud-LLM. No on-device tiered cognition | | **2** | Voice-Commanded Squad Tactics | Phase 78 is modding-only. No gameplay voice control | | **3** | NPC Self-Actuation & Emergent Agency | Behavior trees ≠ genuine emergent agency | | **4** | Unified Cognitive Architecture | No single perception→cognition→action→memory pipeline | | **5** | Voice-as-First-Class Gameplay Input | No NPC voice interaction during actual gameplay | | **6** | Procedural City Interior & Growth | Phase 79.27 has traffic/districts but no building interiors or city evolution | | **7** | Voice-Controlled Combat AI & Formations | Phase 43 has basic combat. No tactical squad voice control | --- ## Industry SOTA Analysis (2025-2026) ### What GTA 6 Is Actually Doing (Patent US11684855B2 + Analysis) Rockstar's approach is **NOT LLM-based**. Based on the patent filing and external technical analysis (note: some specifics below are derived from industry analysis of multiple Rockstar/Take-Two patents and may include extrapolation beyond the single patent's text): - **Server-side batch processing** of 10,000+ NPCs across 64-core CPU clusters - **Lightweight neural networks** (10-input, 4-output) per NPC for driving decisions - **Personality vectors** — 5-7 parameters on 0.0-1.0 scales (aggressiveness, patience, weather sensitivity, risk tolerance) - **Adaptive AI LOD**: <100m full personality modeling, 100-500m simplified behavior trees, >500m statistical movement patterns - **Scale**: 200x improvement over traditional NPC counts (10,000+ vs ~50) - **Bandwidth**: 2KB/s per nearby NPC, 8KB/s in heavy traffic, 4:1 compression - **Response time**: 16ms average, 99.9% uptime **Key Insight**: GTA 6's NPCs are NOT smart. They're **numerous and personality-consistent** using lightweight neural networks. The illusion of life comes from scale + personality variety, not deep cognition. ### Where We Can Surpass GTA 6 GTA 6 chose **breadth over depth** — many NPCs with shallow cognition. We can have **both** by implementing a cognitive LOD system that transitions seamlessly: ``` Distance │ GTA 6 Approach │ Oshun Approach ────────────┼──────────────────────────┼───────────────────────────────────── >1km │ Statistical │ Statistical + alibi generation 100m-1km │ Behavior trees │ Personality-vector neural net (GTA6-class) 15-100m │ Full personality NN │ On-device 2B SLM with personality lock 0-15m │ Full personality NN │ On-device 4-8B SLM with full memory/emotion Interacting │ Scripted dialogue tree │ Full LLM cognition with episodic memory Voice cmd │ N/A │ Natural language understanding + action ``` This means every NPC in the world has **GTA 6-class** behavior as a baseline, but NPCs the player approaches have **Stanford Generative Agent-class** memory and reflection, and NPCs the player talks to have **full conversational depth**. Nobody has shipped this fusion. ### NVIDIA ACE Production Architecture (2025) The most mature production pipeline for AI NPCs: - **On-device models**: Nemotron-Mini-4B (~2GB VRAM), Mistral-NeMo-Minitron 2B/4B/8B - **Pipeline**: Riva ASR → SLM reasoning → Riva TTS → Audio2Face-3D → UE5 - **Perception**: NemoAudio-4B (soundscape), Parakeet-CTC-XXL (multilingual STT), NemoVision-4B (spatial understanding) - **Decision rate**: 8-13 micro-decisions/second (based on esports research) - **Shipped**: PUBG Ally (8B companion), NARAKA (on-device teammates), inZOI ### Google SIMA 2 (DeepMind) The most technically ambitious game agent: - Foundation: Gemini Flash-Lite fine-tuned on embodied gameplay - Input: 720p RGB frames + language + prior responses - Output: Unified token stream → keyboard/mouse actions + dialogue + reasoning - Hierarchical: Gemini Pro for high-level goals, SIMA 2 for visuomotor control - Self-improvement: Agent generates its own tasks and rewards - Near-human performance in interaction, menu use, object management ### Stanford Generative Agents (Smallville) The foundational NPC memory architecture: - **Memory stream**: All experiences logged with timestamp, importance (1-10), last access time - **Retrieval**: `score = α×recency + β×importance + γ×relevance` (embedding cosine similarity) - **Reflection**: Triggered when cumulative importance exceeds threshold → generates insights → stored back as higher-level memories - **Planning**: Daily plans generated each morning → hourly → 5-15 min actions → revised reactively when unexpected events occur - **Emergent result**: From "one agent wants to throw a party," agents autonomously spread invitations, asked each other on dates, coordinated arrival ### F.A.C.U.L. (Tencent) — SOTA Voice-Commanded Game AI The most technically detailed voice-command system for games: - **Dual-model routing**: BERT-FID (104M params, 74.9% accuracy, 412ms) for high-confidence commands → Qwen2-1.5B LLM planner for low-confidence - **Scene recognition**: Fine-tuned CLIP (0.908 F1, 17K+ items) + entity metadata (coordinates, orientations, cover points) - **End-to-end**: 87.2% accuracy, 613ms average response time - **Total footprint**: ~1.6B parameters ### Dwarf Fortress — Deepest Social Simulation - 51 personality facets on 0-100 scales (mapped to NEO PI-R Big Five) - 7 intensity levels with realistic distribution (78% neutral, 0.4% extreme) - 17 facets contribute to grudge formation (>20 point difference) - Memories modify facets over time (personality drift) - Behavioral triggers: ANGER_PROPENSITY 91-100 → tantrum/berserk likelihood - Philosophy: Simulate individual elements, not systems. Emergence arises from autonomous agents exercising free will within systemic constraints. ### Additional SOTA References **Unreal Engine 5 MassAI & Smart Objects**: UE5's Mass Entity framework is the primary competitor architecture for large-scale NPC simulation. Smart Objects provide context-aware NPC-world interaction — an NPC "discovers" a chair can be sat in via Smart Object metadata. Our unified cognitive loop (Gap 4) must match or exceed this capability while adding SLM reasoning on top. **Ready or Not (2024-2025)**: The most relevant modern SWAT tactical AI game. Its contextual command system (Z-menu changes based on what you point at), door breach sequencing, and room clearing AI directly inform Gap 7's tactical squad design. Our voice command system must match Ready or Not's contextual awareness while replacing the menu with natural language. **Baldur's Gate 3 Companion AI**: BG3's companion system — opinion tracking, approval systems, camp conversations, romance arcs — is the gold standard for personality-driven NPC relationships in shipped games (2023-2025). Our Phase 79.25 relationship system combined with Phase 81 SLM cognition must produce companion interactions at least this rich. **Watch Dogs Legion "Play as Any NPC"**: Legion proved that procedurally generated NPC personalities at city scale (every NPC recruitable with unique backstory, skills, voice) is shippable. This validates our Tier 2 personality NN approach for diverse background NPCs, while we push far deeper with SLM cognition for interactive NPCs. **Convai**: Direct commercial competitor to NVIDIA ACE for AI NPC systems, with conversation, knowledge graphs, embodied actions, and Unreal/Unity integrations. Our system must exceed Convai's capabilities by integrating deeper world simulation (economy, politics, memory) beyond what standalone NPC platforms offer. **NPU-Accelerated Inference (2026+)**: Intel Meteor Lake, AMD Ryzen AI, Qualcomm Snapdragon X, Apple Neural Engine, and rumored PS6/next-Xbox NPUs provide dedicated AI inference silicon. Our on-device SLM architecture (Gap 1) should support NPU offloading as a hardware backend alongside GPU, potentially freeing GPU VRAM entirely for rendering while NPU handles NPC cognition. This is not yet implemented but the architecture should be NPU-ready. **Reinforcement Learning for NPC Training**: An alternative to our teacher- student distillation (Gap 1.3) would be training Tier 2 personality NNs via RL rather than supervised learning from SLM decisions. RL could discover more nuanced behavior policies but requires a reward function design that captures "believable NPC behavior" — a hard problem. Our supervised distillation approach is chosen for predictability and alignment with the SLM's reasoning, but RL fine-tuning of distilled NNs could be explored as a future enhancement. ### Gemma-Based Game NPCs (Gemma3NPC) Open-source proof that on-device NPC brains are practical: - **Model**: gemma-3n-E4B (8B params, 4B effective) fine-tuned with LoRA - **Training cost**: $21.56 total, 7 hours on A100 - **Performance**: RTX 5070 Ti → 74 tok/s, RTX 4070 → 55 tok/s - **Key finding**: Fine-tuned models prioritize character consistency over factual accuracy — exactly what NPCs need --- ## Gap 1: On-Device SLM Cognitive Core (`maya-npc-cognition`) ### What's Missing Phase 40.10 (maya/souls) has cloud-LLM NPC conversations. Phase 79.24 has needs/personality/emotions in Rust. **Nothing bridges these** — there's no on-device inference pipeline that runs SLMs inside the game process, manages VRAM budgets across NPCs, or implements the GTA 6-class personality neural network for the thousands of NPCs that don't need full LLM cognition. ### Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ NPC Cognitive LOD Manager │ │ (Rust, manages which NPCs get which cognition tier) │ ├─────────┬───────────┬───────────┬───────────┬───────────┤ │ Tier 4 │ Tier 3 │ Tier 2 │ Tier 1 │ Tier 0 │ │ Stat │ Schedule │ Person.NN │ 2B SLM │ 4-8B SLM │ │ >1km │ 100m-1km │ 15-100m │ 0-15m │ Interact │ │ ~0 CPU │ 0.001ms │ 0.01ms │ ~15ms GPU │ ~30ms GPU │ │ pop#s │ zone loc │ micro-dec │ full cog │ dialogue │ └─────────┴───────────┴───────────┴───────────┴───────────┘ ``` ### Components **1.1 Personality Neural Network (Tier 2 — GTA 6 Class)** Lightweight neural network running on CPU, one per mid-range NPC: - Input: 12 dimensions — personality vector (5 Big Five), current need state (3 most deficient), environmental context (weather, time, danger level), social proximity - Output: 8 action categories — continue current activity, seek need fulfillment, initiate social, avoid threat, pursue goal, idle/wander, emotional reaction, flee - Training: Supervised on behavior data generated by Tier 0/1 SLM decisions ("teacher-student" distillation from SLM to lightweight NN) - Budget: <0.01ms per NPC, can run 5,000+ per frame on 4 CPU cores - Personality vectors from Phase 79.24.2 facets compressed to 5D Big Five **1.2 On-Device SLM Inference Engine (Tier 1 & 0)** Runs Gemma 4 E2B/E4B or Nemotron-Mini-4B inside the game process: - **VRAM budget manager** — allocates inference capacity across active NPCs based on distance, player attention, narrative importance - **Model pool**: 2B model for Tier 1 (quick decisions, barks, reactions), 4-8B model for Tier 0 (full dialogue, complex reasoning, planning) - **Personality-locked prompts** — each NPC's personality facets, values, goals, and current emotional state injected as system prompt constraining SLM output - **Memory-augmented generation** — RAG retrieval from NPC's memory stream (Phase 79.24.4) injected into context window - **Batch inference** — group multiple NPC queries and process in a single forward pass for throughput (GTA 6 patent technique applied to SLMs) - **Compute-in-Graphics (CIG)** scheduling — run SLM inference on GPU between render passes, synchronized with frame timing (NVIDIA ACE technique) **1.3 Teacher-Student Cognition Distillation** The novel architecture that makes the whole system work: - Full SLM decisions (Tier 0/1) are logged as training data - Periodically, personality NN (Tier 2) is fine-tuned on these decisions - This means Tier 2 NPCs make decisions that are *statistically consistent* with what the SLM would have decided — without SLM cost - When a Tier 2 NPC promotes to Tier 1, their behavior is already coherent because the NN was trained from the SLM **1.4 Cloud Fallback for Extreme Cognition** For complex multi-NPC reasoning (negotiations, political schemes, group planning), a cloud LLM (Claude/GPT-level) is queried: - Latency-tolerant — used for decisions that can wait 1-3 seconds - Multi-NPC context — cloud LLM reasons about multiple NPCs simultaneously - Results cached and distilled back into on-device models - Fallback for hardware without sufficient GPU for on-device SLM ### VRAM Budget (Consumer Hardware) | Hardware | SLM Budget | Tier 0 NPCs | Tier 1 NPCs | Tier 2 NPCs | |----------|-----------|-------------|-------------|-------------| | RTX 4060 (8GB) | 2GB | 1 (4B Q4) | 2 batch (2B Q4) | 5,000+ CPU | | RTX 4070 (12GB) | 4GB | 2 (4B Q4) | 4 batch (2B Q4) | 5,000+ CPU | | RTX 4090 (24GB) | 8GB | 3 (8B Q4) | 8 batch (2B Q4) | 10,000+ CPU | | PS6 (NPU) | Dedicated | 4+ (4B Q4) | 10+ (2B Q4) | 10,000+ CPU | --- ## Gap 2: Voice-Commanded Squad Tactics (`maya-voice-tactics`) ### What's Missing Phase 78 has voice ASR for the modding companion. **Nothing covers in-gameplay voice commands** — talking to your squad, commanding a SWAT team breach, directing NPC allies in real-time combat. This is the Rainbow Six / SOCOM / ARMA dream realized with modern AI. ### Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Player Voice Input │ │ (Microphone → VAD → Wake-Word Optional) │ ├─────────────────────────────────────────────────────────┤ │ On-Device ASR (CarelessWhisper/Gemma 4) │ │ 300ms chunks, streaming, noise-robust │ ├──────────────────────┬──────────────────────────────────┤ │ High-Confidence Path │ Low-Confidence Path │ │ BERT-FID Intent │ SLM Planner (2B) │ │ (104M, <100ms) │ (natural language → actions) │ ├──────────────────────┴──────────────────────────────────┤ │ Scene Understanding (CLIP + Entity DB) │ │ "that door" → world coords via gaze + proximity │ ├─────────────────────────────────────────────────────────┤ │ Tactical Command Resolver │ │ Maps intent → atomic squad actions + formations │ ├─────────────────────────────────────────────────────────┤ │ Multi-Agent Coordinator (ORCA + MARL) │ │ Per-agent behavior tree injection at max priority │ ├─────────────────────────────────────────────────────────┤ │ Squad Execution Layer │ │ Per-NPC autonomous BT with voice-override priority │ ├─────────────────────────────────────────────────────────┤ │ TTS Feedback ("Copy, stacking up on door") │ └─────────────────────────────────────────────────────────┘ Target: <650ms end-to-end (matching F.A.C.U.L. benchmark) Model footprint: ~3-4GB (ASR + intent + SLM + CLIP) ``` ### Command Taxonomy **2.1 Structured Tactical Commands (High-Confidence BERT Path)** 25 intent categories across 3 hierarchy layers: ``` Layer 1 — Team Selection ├── "Alpha" / "Bravo" / "Charlie" / "Everyone" / "You" (nearest) │ Layer 2 — Action Category ├── Movement: move to, hold position, fall in, stack up, breach ├── Combat: engage, suppress, cease fire, weapons free/hold ├── Tactical: flash and clear, smoke, cover, overwatch, snipe ├── Formation: wedge, column, line, diamond, spread, tight ├── Posture: go loud, go quiet, stealth, aggressive, defensive │ Layer 3 — Target/Location ├── Spatial: "that door", "left side", "second floor", "behind us" ├── Entity: "the hostile", "the hostage", "the vehicle" ├── Named: "waypoint alpha", "extraction point", "rally point" ``` **2.2 Natural Language Commands (SLM Path)** For complex or ambiguous commands the BERT classifier can't resolve: - "Okay, I want Alpha team to go around the back and breach from the kitchen while Bravo holds the front door. On my signal." - "Someone cover that rooftop sniper, everyone else push left" - "Hold fire until I take the first shot, then weapons free" The SLM decomposes these into sequences of atomic actions with proper temporal ordering and dependencies (wait_for_signal, execute_after, etc.). **2.3 Spatial Reasoning from Voice** The critical unsolved problem in voice-controlled games: - **Gaze-anchored references**: "that door" resolves to the door the player is looking at (raycasted from camera center) - **Relative spatial terms**: "left flank" = 90° left of squad facing direction, "behind us" = 180° from squad movement vector - **Ordinal references**: "second window from the left" — enumerate visible windows left-to-right from player perspective, select index 2 - **Named landmark references**: "the red building" — CLIP similarity search against visible entities with color/type metadata - **Contextual references**: "the same door we came in" — retrieve from squad's episodic memory the entry point **2.4 Voice Feedback & Acknowledgment** NPCs respond vocally to commands: - "Copy, moving to position" — immediate acknowledgment - "Negative, I'm pinned down" — cannot comply + reason - "Hostile down" — status update - "Alpha in position, standing by" — ready confirmation - "Contact! Three hostiles, ground floor!" — autonomous callout - Personality affects speech style (calm professional vs nervous rookie) ### Hardware Requirements | Component | Model | Size | Latency | |-----------|-------|------|---------| | Wake word | Porcupine custom | <10KB | <30ms | | VAD | Silero VAD | <2MB | <10ms | | ASR | CarelessWhisper medium | ~1.5GB | <200ms | | Intent (fast) | BERT-FID fine-tuned | ~400MB | <100ms | | Intent (complex) | Gemma 4 E2B | ~2.3GB | <300ms | | Scene understanding | Fine-tuned CLIP | ~1GB | <50ms | | TTS (feedback) | Piper TTS | ~100MB | <100ms | | **Total** | | **~5.3GB** | **<650ms** | --- ## Gap 3: NPC Self-Actuation & Emergent Agency (`maya-npc-agency`) ### What's Missing Phase 79.24 has needs and personality. Phase 40.10 has behavior trees and GOAP. But neither implements **genuine emergent agency** — NPCs that form their own goals, make plans, execute them over days/weeks of game time, and surprise the player with decisions that emerge from personality + memory + world state rather than designer-authored behavior trees. ### The Stanford Generative Agents Architecture, Adapted for Real-Time Games **3.1 Autonomous Planning System** Each Tier 0/1 NPC generates plans at multiple time horizons: ``` Long-Term Goals (personality + values + life stage) │ "I want to become the best blacksmith in the region" │ "I want to avenge my brother's death" │ "I want to save enough to open my own shop" │ ├── Monthly Plans (goals + resources + constraints) │ "This month: complete 3 masterwork swords, save 50 gold" │ "This month: train combat skills, gather allies" │ ├── Weekly Plans (monthly plan + opportunities + social) │ "This week: buy steel from the traveling merchant Tuesday" │ "This week: ask the guard captain to spar" │ ├── Daily Schedule (weekly plan + needs + weather + events) │ "Today: work forge 6am-noon, lunch at tavern, buy steel 2pm" │ └── Moment-to-Moment (daily schedule + interrupts + reactions) "Right now: the player just insulted me → react based on personality" ``` Plans are generated by the on-device SLM (Tier 0/1) or the personality NN (Tier 2, following pre-generated schedule templates). Plans are **revised reactively** when: - An unexpected event occurs (fire, combat, player interaction) - A need becomes critical (starving → abandon work to find food) - A social opportunity arises (friend passes by → greet) - A goal is blocked (shop burned down → grief → new plan) **3.2 Reflection & Insight Generation** Adapted from Stanford Generative Agents: - **Trigger**: When cumulative importance of new memories exceeds threshold (configurable, default 150) - **Process**: SLM generates high-level questions about recent experiences → retrieves relevant memories → synthesizes insights - **Storage**: Insights stored back into memory stream at higher importance - **Hierarchy**: observations → insights → beliefs → personality drift Example: ``` Observations: "The player helped me carry supplies" (importance 4) "The player defended me from bandits" (importance 7) "The player gave me a healing potion" (importance 5) Reflection trigger: cumulative = 16 > threshold 15 Question generated: "What is my relationship with this player?" Insight generated: "The player has been consistently helpful and protective. I trust them and would help them in return." Stored as: belief (importance 8, type: relationship_assessment) ``` **3.3 Emergent Goal Generation** NPCs generate new goals from personality + memory + world state: - High GREED + observing wealthy merchant → "I want what they have" → theft or entrepreneurship depending on RESPONSIBILITY - High KNOWLEDGE + discovering ancient ruins → "I must study this" → expedition planning - REVENGE thought from brother's murder + sufficient BRAVERY → plan attack - Low NEEDS satisfaction + unemployment → desperation → crime or migration This produces **genuine emergent narrative** — the NPC's story isn't authored by a designer, it emerges from the interaction of personality, circumstances, and memory. **3.4 Multi-NPC Emergent Cooperation** Groups of NPCs independently deciding to cooperate: - Two NPCs with shared enemy + sufficient trust → alliance formation - Village of NPCs facing famine → collective migration decision - Criminal NPCs with complementary skills → gang formation - Scholarly NPCs sharing research interests → study group This uses the relationship graph (Phase 79.25) + personality compatibility + shared goals. The SLM at Tier 0/1 evaluates whether cooperation serves the NPC's goals and decides whether to propose/accept collaboration. --- ## Gap 4: Unified Cognitive Architecture (`maya-cognitive-core`) ### What's Missing The existing plans have separate systems — behavior trees (Phase 43), needs (79.24), memory (79.24.4), personality (79.24.2), emotions (79.24.3), social (79.25), LLM dialogue (40.10). **Nothing unifies these into a single perception→cognition→action→memory loop** that makes each NPC a coherent cognitive agent rather than a collection of independent systems. ### The Unified Loop Every game tick, each Tier 0/1 NPC runs this cycle: ``` ┌─────────────────────────────────────────────────────┐ │ PERCEIVE │ │ ┌────────────┬────────────┬────────────┐ │ │ │ Vision │ Audio │ Game State │ │ │ │ Entities │ Sounds │ Time/Weather│ │ │ │ in FOV │ in range │ Events │ │ │ │ w/ occl. │ w/ attenu. │ Quests │ │ │ └─────┬──────┴─────┬──────┴─────┬──────┘ │ │ └────────────┴────────────┘ │ │ │ │ │ Attention Filter │ │ (salience scoring, change detection) │ │ │ │ ├─────────────────────┼───────────────────────────────┤ │ REMEMBER │ │ │ │ │ ┌───────────────┼───────────────┐ │ │ │ Retrieve │ Store new │ │ │ │ relevant │ percepts as │ │ │ │ memories │ memories │ │ │ │ (RAG) │ w/importance│ │ │ └───────┬───────┴───────┬───────┘ │ │ │ │ │ │ │ Check reflection trigger │ │ │ (if cumulative importance > θ) │ │ │ │ │ ├─────────────┼───────────────┼────────────────────────┤ │ THINK │ │ │ │ │ Current state: needs + emotions + personality │ │ Context: retrieved memories + current percepts │ │ Active plan: today's schedule + long-term goals │ │ │ │ │ ┌───────────────┼───────────────┐ │ │ │ Does current │ Yes: continue │ │ │ │ plan still │ executing │ │ │ │ make sense? │ │ │ │ │ │ No: replan │ │ │ │ │ (SLM query) │ │ │ └───────────────┴───────────────┘ │ │ │ │ ├─────────────────────┼────────────────────────────────┤ │ ACT │ │ │ │ │ Selected action → Behavior Tree execution │ │ Voice override? → Inject high-priority node │ │ Emotional state → Modulate animation/voice │ │ Social context → Adjust interaction style │ │ │ │ └─────────────────────┼────────────────────────────────┘ │ External world effects (movement, speech, interaction) ``` ### Integration with Existing Systems | Existing System | Role in Unified Architecture | |----------------|------------------------------| | Phase 43 behavior trees | ACT layer — execution of decided actions | | Phase 43 navmesh/pathfinding | ACT layer — movement execution | | Phase 79.24.1 needs | THINK layer — urgency signals | | Phase 79.24.2 personality | THINK layer — decision weights | | Phase 79.24.3 emotions | THINK layer — mood modulation of decisions | | Phase 79.24.4 memory | REMEMBER layer — episodic retrieval | | Phase 79.25 relationships | THINK layer — social context | | Phase 79.31 NPC-NPC interaction | ACT layer — social action execution | | Phase 40.10 LLM integration | THINK layer — SLM/cloud reasoning | | Gap 1 SLM engine | THINK layer — on-device inference | | Gap 2 voice commands | PERCEIVE layer — player voice input | ### Attention System NPCs can't process everything — attention budget: - **Salience scoring**: loud sound > movement > static objects - **Change detection**: new entity in FOV scores higher than known entity - **Emotional relevance**: feared entity scores higher, loved entity scores higher - **Goal relevance**: entities related to active plan score higher - **Budget**: Process top 5-10 percepts per tick, ignore rest - **Surprise**: Unexpected percepts (explosion, player appearing) force attention regardless of budget --- ## Gap 5: Voice-as-First-Class Gameplay Input (`maya-voice-gameplay`) ### What's Missing Phase 78's voice pipeline is exclusively for the modding companion. **No existing phase covers players talking to NPCs during actual gameplay** — the ability to walk up to any NPC and have a natural conversation, negotiate with faction leaders using your actual voice, interrogate prisoners, or sweet-talk a merchant for a better price. ### System Design **5.1 Conversational NPC Interaction** Any Tier 0 NPC can engage in voice conversation: ``` Player speaks → ASR (shared with Gap 2 pipeline) │ ├── Is player addressing an NPC? (gaze + proximity + "hey" trigger) │ │ │ └── NPC Dialogue Mode activated │ │ │ ├── Player utterance → NPC's SLM with full context: │ │ • NPC personality, emotions, current activity │ │ • Relationship with player (memory) │ │ • Current world state, time, location │ │ • Active quests, knowledge, rumors │ │ • Conversation history │ │ │ ├── SLM generates response text + emotion + action intent │ │ │ ├── TTS synthesizes voice with personality-matched timbre │ │ • Personality affects: pace, pitch, vocabulary │ │ • Emotion affects: tremor, volume, pauses │ │ │ ├── Audio2Face drives facial animation from speech │ │ │ └── NPC may take actions based on conversation: │ • Give information, open shop, attack, flee │ • Update opinion of player, spread gossip │ • Agree to quest, refuse request, negotiate │ └── Is player giving a squad command? (→ Gap 2 pipeline) ``` **5.2 Voice Negotiation** Players can negotiate using natural language: - **Trade**: "Can you do 30 gold instead of 50?" → NPC evaluates based on greed, player reputation, item value, current need for gold - **Persuasion**: "The bandits are getting worse — we need your militia" → NPC evaluates based on bravery, community values, assessment of threat - **Intimidation**: "You don't want to make an enemy of me" → NPC evaluates based on player reputation, own courage, available backup - **Deception**: "I'm a merchant guild inspector" → NPC evaluates based on player appearance, known facts, suspicion personality trait Each negotiation leverages the NPC's full personality, memory, and emotional state. A greedy NPC with low trust of the player responds differently than a generous NPC who remembers the player helped them. **5.3 Voice-Driven Emergent Narrative** Players can create emergent stories through voice: - Tell an NPC a lie → it enters their memory → they gossip it → rumor spreads (integrates with Phase 79.34 Information Propagation) - Convince an NPC to change careers → they actually do it (integrates with Phase 79.26 Occupation system) - Persuade a faction leader to declare war → political consequences (integrates with Phase 79.36 Political Simulation) - Promise to return with supplies → NPC remembers and expects delivery → trust impact if player doesn't follow through **5.4 Wake-Word and Activation Modes** Three voice input modes: 1. **Push-to-talk**: Traditional, always available 2. **Wake word**: "Hey [NPC name]" activates conversation with specific NPC (Picovoice Porcupine, <10KB per keyword, <30ms) 3. **Proximity auto-listen**: When very close to an NPC in conversation mode, mic is always active (with clear UI indicator) --- ## Gap 6: Procedural City Interior & Urban Evolution (`maya-city-evolution`) ### What's Missing Phase 79.27 covers traffic, transit, emergency services, and district personality. Phase 79.38 covers environmental sound propagation. But **no phase covers**: - Procedural generation of building interiors (furnished rooms, not empty shells) - City growth and change over time (construction, gentrification, decay) - Infrastructure systems that affect gameplay (power outages, water supply) - Dynamic construction sites with NPC workers building new structures ### System Design **6.1 Procedural Building Interiors** When a player enters any building, the interior is procedurally generated: - **Shape grammar** for room layout: building footprint → room subdivision based on building type (residential: bedroom, kitchen, living; commercial: shop floor, storage, office; public: lobby, offices, conference) - **Furnishing system**: room type → furniture placement using constraint solving (bed against wall, desk near window, appropriate clearance for doors) - **Occupation-driven decoration**: scholarly NPC → books, scrolls, ink stains; warrior NPC → weapon racks, armor stands; merchant → displayed wares, accounting books (connects to Phase 79.26) - **Wealth-driven quality**: NPC income determines furnishing quality, repair state, decoration density - **Personality-driven personalization**: messy NPC → cluttered; organized NPC → neat; artistic NPC → paintings on walls - **Persistent interiors**: Once generated, interior state is saved and persists. NPCs move furniture, add/remove items over time. **6.2 Urban Evolution** Cities change over real gameplay time: - **Construction**: New buildings appear through construction process — scaffolding → framework → walls → roof → finishing → occupancy. Construction crew NPCs visibly work the site. Duration: days to weeks of game time. - **Gentrification**: High-wealth district expanding into low-wealth areas → old buildings demolished → new luxury buildings → displaced residents move to cheaper districts - **Decay**: Abandoned buildings deteriorate — broken windows, graffiti, roof leaks, structural damage. Homeless NPCs may squat. - **Repair**: Owner NPCs commission repairs. Player can affect through quests. - **Disaster recovery**: After fire/war, damaged areas rebuild over time with NPC construction crews. **6.3 Infrastructure Systems** - **Power**: Generator/windmill/waterwheel networks. Power outages affect street lights (dark at night), forge temperatures, water pumps. - **Water**: Aqueduct/well/pump systems. Water shortage affects NPC hygiene need satisfaction, crop growth, fire response capability. - **Sewage**: Affects disease spread risk, NPC complaints, district desirability. - **Road maintenance**: Potholes develop with traffic, affect vehicle speed. NPC work crews repair roads. --- ## Gap 7: Voice-Controlled Combat AI & Formations (`maya-combat-voice`) ### What's Missing Phase 43 has behavior trees and pathfinding. Phase 79.1 has Euphoria-class ragdoll. But **no phase covers tactical combat AI with voice control** — the ability to command a squad like Rainbow Six, with NPCs that understand tactical concepts (breach, clear, cover, suppress, flank) and execute them with real military tactics. ### Command Vocabulary ``` MOVEMENT COMMANDS ├── "Move to [location]" — standard movement ├── "Stack up on [door/window]" — form breach line ├── "Hold position" — stay and defend current location ├── "Fall in" — return to formation near player ├── "Advance" — move forward maintaining formation ├── "Retreat to [location]" — fall back to position ├── "Flank left/right" — move to flanking position ├── "Go around back" — find rear entrance/approach BREACH COMMANDS ├── "Breach and clear" — explosive/kick door, enter, clear room ├── "Flash and clear" — flashbang then enter ├── "Smoke and enter" — smoke grenade then enter ├── "Open and hold" — open door without entering, cover from outside ├── "Breach on my mark" — prepare breach, wait for "Go!" command COMBAT COMMANDS ├── "Engage" / "Weapons free" — fire at will on hostiles ├── "Hold fire" / "Weapons hold" — cease fire, maintain position ├── "Suppress [target/area]" — sustained fire on area ├── "Take the shot" — sniper/marksman engage specific target ├── "Cease fire" — stop shooting immediately ├── "Cover me" — provide suppressive fire while player moves FORMATION COMMANDS ├── "Wedge formation" — V-shape, balanced offense/defense ├── "Column" / "Single file" — line behind player ├── "Line abreast" — spread horizontal line ├── "Diamond" — 360° coverage formation ├── "Tight up" — close spacing for indoor ├── "Spread out" — wide spacing for open areas ├── "Bounding overwatch" — alternating advance with cover POSTURE COMMANDS ├── "Go loud" — drop stealth, prioritize firepower ├── "Go quiet" / "Stealth" — suppress weapons, CQC preferred ├── "Aggressive" — push forward, overwhelm ├── "Defensive" — hold ground, conserve ammo ├── "Non-lethal" — tasers/batons only (for hostage situations) ``` ### Squad AI Architecture **7.1 Autonomous Tactical Behavior (Between Commands)** When the player hasn't given a command, squad NPCs act autonomously: - **Threat assessment**: Continuously evaluate nearby threats, assign engagement priority - **Cover seeking**: Always maintain proximity to cover, peek/lean to engage - **Mutual support**: Maintain firing lanes that don't cross friendly positions - **Ammo management**: Call out low ammo, request resupply - **Casualty response**: Drag wounded teammate to cover, apply first aid - **Noise discipline**: In stealth mode, avoid unnecessary noise, whisper comms - **Callouts**: "Contact! Two hostiles, ground floor, left side!" — automatic threat reporting with accurate spatial descriptors **7.2 Voice Command Priority System** ``` Priority 5 (Maximum): Player voice command — always overrides Priority 4: Self-preservation — under fire, take cover first Priority 3: Team safety — rescue downed teammate Priority 2: Standing order — last player command (e.g., "hold this position") Priority 1: Autonomous tactical behavior Priority 0: Idle/formation following ``` When a voice command arrives, it injects a Priority 5 behavior tree node that overrides current behavior. The NPC acknowledges verbally and executes. If execution is impossible (blocked path, no ammo), the NPC reports the problem verbally. **7.3 Multi-Agent Coordination** When the player gives a team command: - **Formation control**: ORCA/RVO for collision avoidance, Boids for cohesion, named formation templates for structured positioning - **Breach synchronization**: When "breach on my mark" is given, all stack members enter ready state. On "go!", synchronized entry with <100ms delta between first and last entry. - **Fire sector assignment**: Each team member automatically assigns a sector to cover based on position and threat direction. No two members cover the same sector. Dynamic reassignment when teammates move or fall. - **Bounding overwatch execution**: Team automatically splits into cover group and movement group, alternating advance. Player voice commands set pace and direction. **7.4 Personality-Driven Combat Behavior** Squad members aren't interchangeable — personality affects combat: - **High BRAVERY**: Pushes aggressively, volunteers for point position, less likely to retreat - **High CAUTIOUSNESS**: Takes extra time to clear corners, double-checks rooms, slower but more thorough - **High AGGRESSION**: Prefers close engagement, may push too far ahead - **Low DISCIPLINE**: Might fire before "weapons free," slower to cease fire - **VETERAN vs ROOKIE**: Veterans peek corners smoothly, transition between positions fluidly. Rookies are jerky, take longer to set up, but improve over time. - **LOYALTY**: High-loyalty NPCs prioritize player safety, will take bullets ### Integration with Existing Systems | Voice-Tactics System | Integrates With | |---------------------|-----------------| | ASR pipeline | Phase 78.1 (shared voice recognition) | | Pathfinding | Phase 43.1 navmesh + cover point graph | | Behavior trees | Phase 43.1.2 with tactical extensions | | NPC personality | Phase 79.24.2 personality facets | | NPC emotions | Phase 79.24.3 emotional state (fear suppresses aggression) | | Euphoria ragdoll | Phase 79.1 for bullet impacts, breaching physics | | Animation | Phase 79.17 facial, Phase 79.2 motion matching for tactical moves | | Audio | Phase 79.38 sound propagation for enemy detection | | Memory | Phase 79.24.4 squad remembers what worked in previous encounters | --- ## Cross-Cutting Concerns ### Performance Budget All 7 gap systems must fit within a strict per-frame budget: | System | CPU Budget | GPU Budget | |--------|-----------|-----------| | Gap 1: Cognitive LOD Manager | 2ms | 4ms (SLM inference) | | Gap 2: Voice Command Pipeline | 0.5ms | 1ms (intent classification) | | Gap 3: NPC Agency (planning/reflection) | 1ms amortized | 2ms amortized (SLM) | | Gap 4: Cognitive Loop (Tier 0 NPCs) | 1ms per NPC | Shared with Gap 1 | | Gap 5: Voice Gameplay | 0.5ms | Shared with Gap 2 | | Gap 6: City Interior Generation | 5ms (on enter) | 0 (uses existing renderer) | | Gap 7: Combat Tactics | 2ms | 0 (CPU-bound) | | **Total** | **~7ms** | **~7ms** | | **Available (16.6ms @60fps)** | **8ms** | **8ms** | ### Technology Stack | Component | Technology | Rationale | |-----------|-----------|-----------| | Cognitive core | Rust crate | <1ms per NPC, SIMD personality NN | | SLM inference | Rust (llama.cpp via FFI) | On-device 2-8B models | | Voice ASR | Rust (whisper.cpp via FFI) | Streaming, noise-robust | | Intent classification | Rust (ONNX Runtime) | BERT-FID at <100ms | | Scene understanding | Rust (ONNX Runtime) | CLIP inference | | Memory/RAG | TypeScript service | E5-Large embeddings, vector search | | Planning/Reflection | TypeScript orchestration | SLM prompt management | | Cloud LLM fallback | TypeScript | Anthropic/Google API calls | | City generation | Rust crate | WFC + shape grammar | | Formation control | Rust crate | ORCA + Boids, <0.01ms per agent | ### Suggested TODOS Phase Structure These 7 gaps could be organized as: ``` Phase 81: NPC Cognitive Core & On-Device SLM (Gap 1 + 4) — 140 tasks 81.1 — Personality Neural Network (Tier 2 lightweight brain) 81.2 — On-Device SLM Inference Engine (llama.cpp integration) 81.3 — VRAM Budget Manager & CIG Scheduling 81.4 — Teacher-Student Cognition Distillation 81.5 — Unified Cognitive Loop (perceive→remember→think→act) 81.6 — Attention System & Salience Scoring 81.7 — Cloud LLM Fallback & Hybrid Reasoning 81.8 — Cognitive LOD Manager (tier assignment, transitions, budgets) Phase 82: Voice-as-First-Class Gameplay Input (Gap 2 + 5 + 7) — 145 tasks 82.1 — On-Device ASR Pipeline (CarelessWhisper/Gemma integration) 82.2 — Tactical Intent Classification (BERT-FID) 82.3 — Natural Language Command Decomposition (SLM planner) 82.4 — Spatial Reasoning from Voice (gaze + CLIP + entity DB) 82.5 — Conversational NPC Interaction (voice dialogue mode) 82.6 — Voice Negotiation & Persuasion System 82.7 — Squad Tactical Command Execution 82.8 — Multi-Agent Formation Control (ORCA + Boids) 82.9 — Combat AI with Personality Variation & Voice Override Priority 82.10 — TTS Feedback & Personality-Matched Voice 82.11 — Wake Word & Activation Modes Phase 83: NPC Emergent Agency & Living World Evolution (Gap 3 + 6) — 128 tasks 83.1 — Autonomous Planning System (multi-horizon goals) 83.2 — Reflection & Insight Generation 83.3 — Emergent Goal Generation from Personality + Memory 83.4 — Multi-NPC Emergent Cooperation 83.5 — Procedural Building Interiors (shape grammar + furnishing) 83.6 — Urban Evolution (construction, decay, gentrification) 83.7 — Infrastructure Systems (power, water, sewage, roads) 83.8 — NPC Lifecycle & Skill Progression (birth, aging, death, skills) 83.9 — Voice-Driven Emergent Narrative Integration ``` --- ## What Makes This Beat GTA 6 and Everything Else | Dimension | GTA 6 | Cyberpunk 2077 | Dwarf Fortress | BG3 | Sims 4 | **Oshun/Maya** | |-----------|-------|---------------|----------------|-----|---------|----------------| | NPC count | 10,000+ | ~200 | ~200 | ~20 | ~8 | **10,000+ with cognitive LOD** | | NPC cognition | Lightweight NN | Scripted sched. | ~50 facets | Script+approval | Needs+traits | **~50 facets + SLM + memory + reflection** | | Voice interaction | None | None | None | None | None | **Full natural language voice with any NPC** | | Voice squad control | None | None | None | None | None | **Full tactical voice control (R6/SOCOM class)** | | Emergent behavior | Low | Low | High | Medium | Medium | **Stanford Generative Agent-class at DF depth** | | City alive | Yes (scale) | Day/night sched. | N/A | N/A | No | **Scale + interiors + evolution + infrastructure** | | NPC memory | None | None | Limited | Limited | None | **Full episodic + semantic + reflective hierarchy** | | NPC relationships | None | Scripted | Deep | Deep+scripted | Scripted | **Deep + emergent + LLM-enriched** | | NPC lifecycle | None | None | Full | None | Aging | **Birth, aging, death, inheritance, skill growth** | | Economy | Static | Static | Agent-based | Static | Static | **Agent-based + personality-driven decisions** | | Political sim | None | None | Basic | None | None | **CK3-class with NPC personality integration** | **The fundamental insight**: Nobody has combined: 1. GTA 6's **scale** (10K+ NPCs via lightweight NNs) 2. Dwarf Fortress's **depth** (51+ personality facets, emergent social dynamics) 3. Stanford Generative Agents' **memory and reflection** 4. NVIDIA ACE's **on-device SLM cognition** 5. F.A.C.U.L.'s **voice command architecture** 6. SOCOM/Rainbow Six's **tactical squad commands** ...into a single coherent system. Each exists in isolation. The integration IS the innovation. --- ## Dependencies ``` Phase 43 (Neith AI Runtime) ─────────┐ Phase 40.10 (maya/souls) ────────────┤ Phase 78.1 (Voice ASR) ─────────────┤ Phase 79.1 (Euphoria ragdoll) ──────┤ Phase 79.11 (Crowd simulation) ─────┤ Phase 79.24 (NPC Life Simulation) ──┼──→ Phase 81 (Cognitive Core) Phase 79.25 (NPC Relationships) ────┤ │ Phase 79.26 (NPC Occupations) ──────┤ ├──→ Phase 82 (Voice Gameplay) Phase 79.27 (City Infrastructure) ──┤ │ │ Phase 79.28 (Economy) ──────────────┤ ├──→ Phase 83 (Emergent Agency) Phase 79.29 (Justice System) ───────┤ Phase 79.31 (NPC-NPC Interaction) ──┤ Phase 79.32 (Simulation LOD) ───────┤ Phase 79.33 (World History) ────────┤ Phase 79.34 (Info Propagation) ─────┘ ``` **Note on Phase 79.11 (Crowd Simulation) integration:** The cognitive LOD system (Phase 81.8) and the crowd simulation LOD (Phase 79.11) must share the same NPC entity pool. Tier 3/4 (statistical/schedule) NPCs from the cognitive system ARE the Tier 3/4 (Bulk/Impostor) crowd agents from Phase 79.11. When a crowd NPC promotes to Tier 2+ cognition, it simultaneously promotes in the crowd rendering LOD. The cognitive LOD manager (81.8) and the crowd LOD system (79.11) must use a shared priority scoring function to prevent conflicting tier assignments. **Note on multiplayer:** These phases are designed for single-player and cooperative multiplayer. For MMO contexts (Phase 80), additional work is needed: NPC cognitive state must be server-authoritative, voice commands from multiple players to the same NPC need arbitration, and NPC memory of different players must be isolated. Phase 80 integration should be addressed as a follow-on phase. --- ## Open Questions for Decision 1. **Primary on-device model**: Gemma 4 E4B (Google, Apache 2.0, native audio input) vs Nemotron-Mini-4B (NVIDIA, optimized for role-play) vs fine-tuned Llama (Meta)? Recommendation: Gemma 4 for audio input capability, Nemotron for NPC dialogue quality. Support both. 2. **ASR model**: CarelessWhisper (streaming Whisper variant, proven) vs Gemma 4 E2B (native audio, smaller but newer)? Recommendation: CarelessWhisper for initial implementation (more mature streaming support), Gemma 4 as future option. 3. **Voice command vs voice conversation routing**: How does the system decide whether the player is giving a tactical command or talking to an NPC? Recommendation: Context-based — if the player is in a squad tactical scenario and looking at teammates, route to command pipeline. If looking at a civilian NPC, route to conversation. Player can override with push-to-talk modes. 4. **VRAM allocation priority**: When GPU memory is limited, should the system prioritize more Tier 1 NPCs (wider but shallower cognition) or fewer Tier 0 NPCs (deeper cognition for key NPCs)? Recommendation: Quality presets. "Immersion" preset favors breadth (many thinking NPCs). "Story" preset favors depth (fewer but deeply realized NPCs). 5. **SLM fine-tuning strategy**: Fine-tune one model per personality archetype (warrior, scholar, merchant) or one model with personality-conditioned prompts? Recommendation: Single model with personality prompts for flexibility. LoRA adapters for extreme specialization (e.g., unique companion characters).