Living World & Autonomous NPC SOTA — Beyond GTA 6

Executive Summary#

This proposal identifies 7 critical system gaps between the Oshun/Maya engine ecosystem's existing plans and what would be required to build the most advanced living world and autonomous NPC system ever created — surpassing GTA 6 (Rockstar), Cyberpunk 2077 (CDPR), RDR2, The Sims, Dwarf Fortress, and everything publicly known from NVIDIA ACE, Google SIMA 2, and Stanford Generative Agents.

The existing TODOS already cover substantial ground:

• Phase 40.10 — maya/souls LLM NPC foundations (all marked done)
• Phase 43 — Neith AI Runtime: navmesh, behavior trees, state machines (done)
• Phase 72 — Neural rendering, game engine SOTA gap closure
• Phase 78 — Voice ASR pipeline for modding companion
• Phase 79.11 — Crowd simulation at 10K+ NPCs
• Phase 79.24-36 — NPC life simulation, relationships, occupations, cities, economy, law, politics, gossip, animals, culture, history
• Phase 80 — Social fabric for MMO community

This proposal fills the unfilled gaps — the systems that would make Oshun's NPCs not just technically competitive but categorically different from anything that exists.

What This Proposal Covers (The 7 Gaps)#

Industry SOTA Analysis (2025-2026)#

What GTA 6 Is Actually Doing (Patent US11684855B2 + Analysis)#

Rockstar's approach is NOT LLM-based. Based on the patent filing and external technical analysis (note: some specifics below are derived from industry analysis of multiple Rockstar/Take-Two patents and may include extrapolation beyond the single patent's text):

• Server-side batch processing of 10,000+ NPCs across 64-core CPU clusters
• Lightweight neural networks (10-input, 4-output) per NPC for driving decisions
• Personality vectors — 5-7 parameters on 0.0-1.0 scales (aggressiveness, patience, weather sensitivity, risk tolerance)
• Adaptive AI LOD: <100m full personality modeling, 100-500m simplified behavior trees, >500m statistical movement patterns
• Scale: 200x improvement over traditional NPC counts (10,000+ vs ~50)
• Bandwidth: 2KB/s per nearby NPC, 8KB/s in heavy traffic, 4:1 compression
• Response time: 16ms average, 99.9% uptime

Key Insight: GTA 6's NPCs are NOT smart. They're numerous and personality-consistent using lightweight neural networks. The illusion of life comes from scale + personality variety, not deep cognition.

Where We Can Surpass GTA 6#

GTA 6 chose breadth over depth — many NPCs with shallow cognition. We can have both by implementing a cognitive LOD system that transitions seamlessly:

Distance    │ GTA 6 Approach           │ Oshun Approach
────────────┼──────────────────────────┼─────────────────────────────────────
>1km        │ Statistical              │ Statistical + alibi generation
100m-1km    │ Behavior trees           │ Personality-vector neural net (GTA6-class)
15-100m     │ Full personality NN      │ On-device 2B SLM with personality lock
0-15m       │ Full personality NN      │ On-device 4-8B SLM with full memory/emotion
Interacting │ Scripted dialogue tree   │ Full LLM cognition with episodic memory
Voice cmd   │ N/A                      │ Natural language understanding + action

This means every NPC in the world has GTA 6-class behavior as a baseline, but NPCs the player approaches have Stanford Generative Agent-class memory and reflection, and NPCs the player talks to have full conversational depth. Nobody has shipped this fusion.

NVIDIA ACE Production Architecture (2025)#

The most mature production pipeline for AI NPCs:

• On-device models: Nemotron-Mini-4B (~2GB VRAM), Mistral-NeMo-Minitron 2B/4B/8B
• Pipeline: Riva ASR → SLM reasoning → Riva TTS → Audio2Face-3D → UE5
• Perception: NemoAudio-4B (soundscape), Parakeet-CTC-XXL (multilingual STT), NemoVision-4B (spatial understanding)
• Decision rate: 8-13 micro-decisions/second (based on esports research)
• Shipped: PUBG Ally (8B companion), NARAKA (on-device teammates), inZOI

Google SIMA 2 (DeepMind)#

The most technically ambitious game agent:

• Foundation: Gemini Flash-Lite fine-tuned on embodied gameplay
• Input: 720p RGB frames + language + prior responses
• Output: Unified token stream → keyboard/mouse actions + dialogue + reasoning
• Hierarchical: Gemini Pro for high-level goals, SIMA 2 for visuomotor control
• Self-improvement: Agent generates its own tasks and rewards
• Near-human performance in interaction, menu use, object management

Stanford Generative Agents (Smallville)#

The foundational NPC memory architecture:

• Memory stream: All experiences logged with timestamp, importance (1-10), last access time
• Retrieval: score = α×recency + β×importance + γ×relevance (embedding cosine similarity)
• Reflection: Triggered when cumulative importance exceeds threshold → generates insights → stored back as higher-level memories
• Planning: Daily plans generated each morning → hourly → 5-15 min actions → revised reactively when unexpected events occur
• Emergent result: From "one agent wants to throw a party," agents autonomously spread invitations, asked each other on dates, coordinated arrival

F.A.C.U.L. (Tencent) — SOTA Voice-Commanded Game AI#

The most technically detailed voice-command system for games:

• Dual-model routing: BERT-FID (104M params, 74.9% accuracy, 412ms) for high-confidence commands → Qwen2-1.5B LLM planner for low-confidence
• Scene recognition: Fine-tuned CLIP (0.908 F1, 17K+ items) + entity metadata (coordinates, orientations, cover points)
• End-to-end: 87.2% accuracy, 613ms average response time
• Total footprint: ~1.6B parameters

Dwarf Fortress — Deepest Social Simulation#

• 51 personality facets on 0-100 scales (mapped to NEO PI-R Big Five)
• 7 intensity levels with realistic distribution (78% neutral, 0.4% extreme)
• 17 facets contribute to grudge formation (>20 point difference)
• Memories modify facets over time (personality drift)
• Behavioral triggers: ANGER_PROPENSITY 91-100 → tantrum/berserk likelihood
• Philosophy: Simulate individual elements, not systems. Emergence arises from autonomous agents exercising free will within systemic constraints.

Additional SOTA References#

Unreal Engine 5 MassAI & Smart Objects: UE5's Mass Entity framework is the primary competitor architecture for large-scale NPC simulation. Smart Objects provide context-aware NPC-world interaction — an NPC "discovers" a chair can be sat in via Smart Object metadata. Our unified cognitive loop (Gap 4) must match or exceed this capability while adding SLM reasoning on top.

Ready or Not (2024-2025): The most relevant modern SWAT tactical AI game. Its contextual command system (Z-menu changes based on what you point at), door breach sequencing, and room clearing AI directly inform Gap 7's tactical squad design. Our voice command system must match Ready or Not's contextual awareness while replacing the menu with natural language.

Baldur's Gate 3 Companion AI: BG3's companion system — opinion tracking, approval systems, camp conversations, romance arcs — is the gold standard for personality-driven NPC relationships in shipped games (2023-2025). Our Phase 79.25 relationship system combined with Phase 81 SLM cognition must produce companion interactions at least this rich.

Watch Dogs Legion "Play as Any NPC": Legion proved that procedurally generated NPC personalities at city scale (every NPC recruitable with unique backstory, skills, voice) is shippable. This validates our Tier 2 personality NN approach for diverse background NPCs, while we push far deeper with SLM cognition for interactive NPCs.

Convai: Direct commercial competitor to NVIDIA ACE for AI NPC systems, with conversation, knowledge graphs, embodied actions, and Unreal/Unity integrations. Our system must exceed Convai's capabilities by integrating deeper world simulation (economy, politics, memory) beyond what standalone NPC platforms offer.

NPU-Accelerated Inference (2026+): Intel Meteor Lake, AMD Ryzen AI, Qualcomm Snapdragon X, Apple Neural Engine, and rumored PS6/next-Xbox NPUs provide dedicated AI inference silicon. Our on-device SLM architecture (Gap 1) should support NPU offloading as a hardware backend alongside GPU, potentially freeing GPU VRAM entirely for rendering while NPU handles NPC cognition. This is not yet implemented but the architecture should be NPU-ready.

Reinforcement Learning for NPC Training: An alternative to our teacher- student distillation (Gap 1.3) would be training Tier 2 personality NNs via RL rather than supervised learning from SLM decisions. RL could discover more nuanced behavior policies but requires a reward function design that captures "believable NPC behavior" — a hard problem. Our supervised distillation approach is chosen for predictability and alignment with the SLM's reasoning, but RL fine-tuning of distilled NNs could be explored as a future enhancement.

Gemma-Based Game NPCs (Gemma3NPC)#

Open-source proof that on-device NPC brains are practical:

• Model: gemma-3n-E4B (8B params, 4B effective) fine-tuned with LoRA
• Training cost: $21.56 total, 7 hours on A100
• Performance: RTX 5070 Ti → 74 tok/s, RTX 4070 → 55 tok/s
• Key finding: Fine-tuned models prioritize character consistency over factual accuracy — exactly what NPCs need

Gap 1: On-Device SLM Cognitive Core (maya-npc-cognition)#

What's Missing#

Phase 40.10 (maya/souls) has cloud-LLM NPC conversations. Phase 79.24 has needs/personality/emotions in Rust. Nothing bridges these — there's no on-device inference pipeline that runs SLMs inside the game process, manages VRAM budgets across NPCs, or implements the GTA 6-class personality neural network for the thousands of NPCs that don't need full LLM cognition.

Architecture#

┌─────────────────────────────────────────────────────────┐
│                  NPC Cognitive LOD Manager               │
│  (Rust, manages which NPCs get which cognition tier)    │
├─────────┬───────────┬───────────┬───────────┬───────────┤
│ Tier 4  │ Tier 3    │ Tier 2    │ Tier 1    │ Tier 0    │
│ Stat    │ Schedule  │ Person.NN │ 2B SLM    │ 4-8B SLM  │
│ >1km    │ 100m-1km  │ 15-100m   │ 0-15m     │ Interact  │
│ ~0 CPU  │ 0.001ms   │ 0.01ms    │ ~15ms GPU │ ~30ms GPU │
│ pop#s   │ zone loc  │ micro-dec │ full cog  │ dialogue  │
└─────────┴───────────┴───────────┴───────────┴───────────┘

Components#

1.1 Personality Neural Network (Tier 2 — GTA 6 Class)

Lightweight neural network running on CPU, one per mid-range NPC:

• Input: 12 dimensions — personality vector (5 Big Five), current need state (3 most deficient), environmental context (weather, time, danger level), social proximity
• Output: 8 action categories — continue current activity, seek need fulfillment, initiate social, avoid threat, pursue goal, idle/wander, emotional reaction, flee
• Training: Supervised on behavior data generated by Tier 0/1 SLM decisions ("teacher-student" distillation from SLM to lightweight NN)
• Budget: <0.01ms per NPC, can run 5,000+ per frame on 4 CPU cores
• Personality vectors from Phase 79.24.2 facets compressed to 5D Big Five

1.2 On-Device SLM Inference Engine (Tier 1 & 0)

Runs Gemma 4 E2B/E4B or Nemotron-Mini-4B inside the game process:

• VRAM budget manager — allocates inference capacity across active NPCs based on distance, player attention, narrative importance
• Model pool: 2B model for Tier 1 (quick decisions, barks, reactions), 4-8B model for Tier 0 (full dialogue, complex reasoning, planning)
• Personality-locked prompts — each NPC's personality facets, values, goals, and current emotional state injected as system prompt constraining SLM output
• Memory-augmented generation — RAG retrieval from NPC's memory stream (Phase 79.24.4) injected into context window
• Batch inference — group multiple NPC queries and process in a single forward pass for throughput (GTA 6 patent technique applied to SLMs)
• Compute-in-Graphics (CIG) scheduling — run SLM inference on GPU between render passes, synchronized with frame timing (NVIDIA ACE technique)

1.3 Teacher-Student Cognition Distillation

The novel architecture that makes the whole system work:

• Full SLM decisions (Tier 0/1) are logged as training data
• Periodically, personality NN (Tier 2) is fine-tuned on these decisions
• This means Tier 2 NPCs make decisions that are statistically consistent with what the SLM would have decided — without SLM cost
• When a Tier 2 NPC promotes to Tier 1, their behavior is already coherent because the NN was trained from the SLM

1.4 Cloud Fallback for Extreme Cognition

For complex multi-NPC reasoning (negotiations, political schemes, group planning), a cloud LLM (Claude/GPT-level) is queried:

• Latency-tolerant — used for decisions that can wait 1-3 seconds
• Multi-NPC context — cloud LLM reasons about multiple NPCs simultaneously
• Results cached and distilled back into on-device models
• Fallback for hardware without sufficient GPU for on-device SLM

VRAM Budget (Consumer Hardware)#

Gap 2: Voice-Commanded Squad Tactics (maya-voice-tactics)#

What's Missing#

Phase 78 has voice ASR for the modding companion. Nothing covers in-gameplay voice commands — talking to your squad, commanding a SWAT team breach, directing NPC allies in real-time combat. This is the Rainbow Six / SOCOM / ARMA dream realized with modern AI.

Architecture#

┌─────────────────────────────────────────────────────────┐
│                Player Voice Input                       │
│     (Microphone → VAD → Wake-Word Optional)             │
├─────────────────────────────────────────────────────────┤
│         On-Device ASR (CarelessWhisper/Gemma 4)         │
│         300ms chunks, streaming, noise-robust            │
├──────────────────────┬──────────────────────────────────┤
│ High-Confidence Path │ Low-Confidence Path              │
│ BERT-FID Intent      │ SLM Planner (2B)                 │
│ (104M, <100ms)       │ (natural language → actions)      │
├──────────────────────┴──────────────────────────────────┤
│           Scene Understanding (CLIP + Entity DB)         │
│     "that door" → world coords via gaze + proximity     │
├─────────────────────────────────────────────────────────┤
│         Tactical Command Resolver                        │
│   Maps intent → atomic squad actions + formations        │
├─────────────────────────────────────────────────────────┤
│         Multi-Agent Coordinator (ORCA + MARL)            │
│   Per-agent behavior tree injection at max priority      │
├─────────────────────────────────────────────────────────┤
│         Squad Execution Layer                            │
│   Per-NPC autonomous BT with voice-override priority     │
├─────────────────────────────────────────────────────────┤
│         TTS Feedback ("Copy, stacking up on door")       │
└─────────────────────────────────────────────────────────┘

Target: <650ms end-to-end (matching F.A.C.U.L. benchmark)
Model footprint: ~3-4GB (ASR + intent + SLM + CLIP)

Command Taxonomy#

2.1 Structured Tactical Commands (High-Confidence BERT Path)

25 intent categories across 3 hierarchy layers:

Layer 1 — Team Selection
├── "Alpha" / "Bravo" / "Charlie" / "Everyone" / "You" (nearest)
│
Layer 2 — Action Category
├── Movement: move to, hold position, fall in, stack up, breach
├── Combat: engage, suppress, cease fire, weapons free/hold
├── Tactical: flash and clear, smoke, cover, overwatch, snipe
├── Formation: wedge, column, line, diamond, spread, tight
├── Posture: go loud, go quiet, stealth, aggressive, defensive
│
Layer 3 — Target/Location
├── Spatial: "that door", "left side", "second floor", "behind us"
├── Entity: "the hostile", "the hostage", "the vehicle"
├── Named: "waypoint alpha", "extraction point", "rally point"

2.2 Natural Language Commands (SLM Path)

For complex or ambiguous commands the BERT classifier can't resolve:

• "Okay, I want Alpha team to go around the back and breach from the kitchen while Bravo holds the front door. On my signal."
• "Someone cover that rooftop sniper, everyone else push left"
• "Hold fire until I take the first shot, then weapons free"

The SLM decomposes these into sequences of atomic actions with proper temporal ordering and dependencies (wait_for_signal, execute_after, etc.).

2.3 Spatial Reasoning from Voice

The critical unsolved problem in voice-controlled games:

• Gaze-anchored references: "that door" resolves to the door the player is looking at (raycasted from camera center)
• Relative spatial terms: "left flank" = 90° left of squad facing direction, "behind us" = 180° from squad movement vector
• Ordinal references: "second window from the left" — enumerate visible windows left-to-right from player perspective, select index 2
• Named landmark references: "the red building" — CLIP similarity search against visible entities with color/type metadata
• Contextual references: "the same door we came in" — retrieve from squad's episodic memory the entry point

2.4 Voice Feedback & Acknowledgment

NPCs respond vocally to commands:

• "Copy, moving to position" — immediate acknowledgment
• "Negative, I'm pinned down" — cannot comply + reason
• "Hostile down" — status update
• "Alpha in position, standing by" — ready confirmation
• "Contact! Three hostiles, ground floor!" — autonomous callout
• Personality affects speech style (calm professional vs nervous rookie)

Hardware Requirements#

Gap 3: NPC Self-Actuation & Emergent Agency (maya-npc-agency)#

What's Missing#

Phase 79.24 has needs and personality. Phase 40.10 has behavior trees and GOAP. But neither implements genuine emergent agency — NPCs that form their own goals, make plans, execute them over days/weeks of game time, and surprise the player with decisions that emerge from personality + memory + world state rather than designer-authored behavior trees.

The Stanford Generative Agents Architecture, Adapted for Real-Time Games#

3.1 Autonomous Planning System

Each Tier 0/1 NPC generates plans at multiple time horizons:

Long-Term Goals (personality + values + life stage)
  │  "I want to become the best blacksmith in the region"
  │  "I want to avenge my brother's death"
  │  "I want to save enough to open my own shop"
  │
  ├── Monthly Plans (goals + resources + constraints)
  │     "This month: complete 3 masterwork swords, save 50 gold"
  │     "This month: train combat skills, gather allies"
  │
  ├── Weekly Plans (monthly plan + opportunities + social)
  │     "This week: buy steel from the traveling merchant Tuesday"
  │     "This week: ask the guard captain to spar"
  │
  ├── Daily Schedule (weekly plan + needs + weather + events)
  │     "Today: work forge 6am-noon, lunch at tavern, buy steel 2pm"
  │
  └── Moment-to-Moment (daily schedule + interrupts + reactions)
        "Right now: the player just insulted me → react based on personality"

Plans are generated by the on-device SLM (Tier 0/1) or the personality NN (Tier 2, following pre-generated schedule templates). Plans are revised reactively when:

• An unexpected event occurs (fire, combat, player interaction)
• A need becomes critical (starving → abandon work to find food)
• A social opportunity arises (friend passes by → greet)
• A goal is blocked (shop burned down → grief → new plan)

3.2 Reflection & Insight Generation

Adapted from Stanford Generative Agents:

• Trigger: When cumulative importance of new memories exceeds threshold (configurable, default 150)
• Process: SLM generates high-level questions about recent experiences → retrieves relevant memories → synthesizes insights
• Storage: Insights stored back into memory stream at higher importance
• Hierarchy: observations → insights → beliefs → personality drift

Example:

Observations: "The player helped me carry supplies" (importance 4)
              "The player defended me from bandits" (importance 7)
              "The player gave me a healing potion" (importance 5)
Reflection trigger: cumulative = 16 > threshold 15
Question generated: "What is my relationship with this player?"
Insight generated: "The player has been consistently helpful and protective.
                    I trust them and would help them in return."
Stored as: belief (importance 8, type: relationship_assessment)

3.3 Emergent Goal Generation

NPCs generate new goals from personality + memory + world state:

• High GREED + observing wealthy merchant → "I want what they have" → theft or entrepreneurship depending on RESPONSIBILITY
• High KNOWLEDGE + discovering ancient ruins → "I must study this" → expedition planning
• REVENGE thought from brother's murder + sufficient BRAVERY → plan attack
• Low NEEDS satisfaction + unemployment → desperation → crime or migration

This produces genuine emergent narrative — the NPC's story isn't authored by a designer, it emerges from the interaction of personality, circumstances, and memory.

3.4 Multi-NPC Emergent Cooperation

Groups of NPCs independently deciding to cooperate:

• Two NPCs with shared enemy + sufficient trust → alliance formation
• Village of NPCs facing famine → collective migration decision
• Criminal NPCs with complementary skills → gang formation
• Scholarly NPCs sharing research interests → study group

This uses the relationship graph (Phase 79.25) + personality compatibility + shared goals. The SLM at Tier 0/1 evaluates whether cooperation serves the NPC's goals and decides whether to propose/accept collaboration.

Gap 4: Unified Cognitive Architecture (maya-cognitive-core)#

What's Missing#

The existing plans have separate systems — behavior trees (Phase 43), needs (79.24), memory (79.24.4), personality (79.24.2), emotions (79.24.3), social (79.25), LLM dialogue (40.10). Nothing unifies these into a single perception→cognition→action→memory loop that makes each NPC a coherent cognitive agent rather than a collection of independent systems.

The Unified Loop#

Every game tick, each Tier 0/1 NPC runs this cycle:

┌─────────────────────────────────────────────────────┐
│                    PERCEIVE                          │
│  ┌────────────┬────────────┬────────────┐           │
│  │   Vision   │   Audio    │ Game State │           │
│  │ Entities   │ Sounds     │ Time/Weather│          │
│  │ in FOV     │ in range   │ Events     │           │
│  │ w/ occl.   │ w/ attenu. │ Quests     │           │
│  └─────┬──────┴─────┬──────┴─────┬──────┘           │
│        └────────────┴────────────┘                   │
│                     │                                │
│              Attention Filter                        │
│        (salience scoring, change detection)          │
│                     │                                │
├─────────────────────┼───────────────────────────────┤
│                  REMEMBER                            │
│                     │                                │
│     ┌───────────────┼───────────────┐                │
│     │   Retrieve    │   Store new   │                │
│     │   relevant    │   percepts as │                │
│     │   memories    │   memories    │                │
│     │   (RAG)       │   w/importance│                │
│     └───────┬───────┴───────┬───────┘                │
│             │               │                        │
│             │    Check reflection trigger             │
│             │    (if cumulative importance > θ)       │
│             │               │                        │
├─────────────┼───────────────┼────────────────────────┤
│                   THINK                              │
│                     │                                │
│     Current state: needs + emotions + personality    │
│     Context: retrieved memories + current percepts   │
│     Active plan: today's schedule + long-term goals  │
│                     │                                │
│     ┌───────────────┼───────────────┐                │
│     │ Does current  │ Yes: continue │                │
│     │ plan still    │ executing     │                │
│     │ make sense?   │               │                │
│     │               │ No: replan    │                │
│     │               │ (SLM query)   │                │
│     └───────────────┴───────────────┘                │
│                     │                                │
├─────────────────────┼────────────────────────────────┤
│                    ACT                               │
│                     │                                │
│     Selected action → Behavior Tree execution        │
│     Voice override? → Inject high-priority node      │
│     Emotional state → Modulate animation/voice       │
│     Social context → Adjust interaction style        │
│                     │                                │
└─────────────────────┼────────────────────────────────┘
                      │
              External world effects
              (movement, speech, interaction)

Integration with Existing Systems#

Attention System#

NPCs can't process everything — attention budget:

• Salience scoring: loud sound > movement > static objects
• Change detection: new entity in FOV scores higher than known entity
• Emotional relevance: feared entity scores higher, loved entity scores higher
• Goal relevance: entities related to active plan score higher
• Budget: Process top 5-10 percepts per tick, ignore rest
• Surprise: Unexpected percepts (explosion, player appearing) force attention regardless of budget

Gap 5: Voice-as-First-Class Gameplay Input (maya-voice-gameplay)#

What's Missing#

Phase 78's voice pipeline is exclusively for the modding companion. No existing phase covers players talking to NPCs during actual gameplay — the ability to walk up to any NPC and have a natural conversation, negotiate with faction leaders using your actual voice, interrogate prisoners, or sweet-talk a merchant for a better price.

System Design#

5.1 Conversational NPC Interaction

Any Tier 0 NPC can engage in voice conversation:

Player speaks → ASR (shared with Gap 2 pipeline)
    │
    ├── Is player addressing an NPC? (gaze + proximity + "hey" trigger)
    │     │
    │     └── NPC Dialogue Mode activated
    │           │
    │           ├── Player utterance → NPC's SLM with full context:
    │           │     • NPC personality, emotions, current activity
    │           │     • Relationship with player (memory)
    │           │     • Current world state, time, location
    │           │     • Active quests, knowledge, rumors
    │           │     • Conversation history
    │           │
    │           ├── SLM generates response text + emotion + action intent
    │           │
    │           ├── TTS synthesizes voice with personality-matched timbre
    │           │     • Personality affects: pace, pitch, vocabulary
    │           │     • Emotion affects: tremor, volume, pauses
    │           │
    │           ├── Audio2Face drives facial animation from speech
    │           │
    │           └── NPC may take actions based on conversation:
    │                 • Give information, open shop, attack, flee
    │                 • Update opinion of player, spread gossip
    │                 • Agree to quest, refuse request, negotiate
    │
    └── Is player giving a squad command? (→ Gap 2 pipeline)

5.2 Voice Negotiation

Players can negotiate using natural language:

• Trade: "Can you do 30 gold instead of 50?" → NPC evaluates based on greed, player reputation, item value, current need for gold
• Persuasion: "The bandits are getting worse — we need your militia" → NPC evaluates based on bravery, community values, assessment of threat
• Intimidation: "You don't want to make an enemy of me" → NPC evaluates based on player reputation, own courage, available backup
• Deception: "I'm a merchant guild inspector" → NPC evaluates based on player appearance, known facts, suspicion personality trait

Each negotiation leverages the NPC's full personality, memory, and emotional state. A greedy NPC with low trust of the player responds differently than a generous NPC who remembers the player helped them.

5.3 Voice-Driven Emergent Narrative

Players can create emergent stories through voice:

• Tell an NPC a lie → it enters their memory → they gossip it → rumor spreads (integrates with Phase 79.34 Information Propagation)
• Convince an NPC to change careers → they actually do it (integrates with Phase 79.26 Occupation system)
• Persuade a faction leader to declare war → political consequences (integrates with Phase 79.36 Political Simulation)
• Promise to return with supplies → NPC remembers and expects delivery → trust impact if player doesn't follow through

5.4 Wake-Word and Activation Modes

Three voice input modes:

1. Push-to-talk: Traditional, always available
2. Wake word: "Hey [NPC name]" activates conversation with specific NPC (Picovoice Porcupine, <10KB per keyword, <30ms)
3. Proximity auto-listen: When very close to an NPC in conversation mode, mic is always active (with clear UI indicator)

Gap 6: Procedural City Interior & Urban Evolution (maya-city-evolution)#

What's Missing#

Phase 79.27 covers traffic, transit, emergency services, and district personality. Phase 79.38 covers environmental sound propagation. But no phase covers:

• Procedural generation of building interiors (furnished rooms, not empty shells)
• City growth and change over time (construction, gentrification, decay)
• Infrastructure systems that affect gameplay (power outages, water supply)
• Dynamic construction sites with NPC workers building new structures

System Design#

6.1 Procedural Building Interiors

When a player enters any building, the interior is procedurally generated:

• Shape grammar for room layout: building footprint → room subdivision based on building type (residential: bedroom, kitchen, living; commercial: shop floor, storage, office; public: lobby, offices, conference)
• Furnishing system: room type → furniture placement using constraint solving (bed against wall, desk near window, appropriate clearance for doors)
• Occupation-driven decoration: scholarly NPC → books, scrolls, ink stains; warrior NPC → weapon racks, armor stands; merchant → displayed wares, accounting books (connects to Phase 79.26)
• Wealth-driven quality: NPC income determines furnishing quality, repair state, decoration density
• Personality-driven personalization: messy NPC → cluttered; organized NPC → neat; artistic NPC → paintings on walls
• Persistent interiors: Once generated, interior state is saved and persists. NPCs move furniture, add/remove items over time.

6.2 Urban Evolution

Cities change over real gameplay time:

• Construction: New buildings appear through construction process — scaffolding → framework → walls → roof → finishing → occupancy. Construction crew NPCs visibly work the site. Duration: days to weeks of game time.
• Gentrification: High-wealth district expanding into low-wealth areas → old buildings demolished → new luxury buildings → displaced residents move to cheaper districts
• Decay: Abandoned buildings deteriorate — broken windows, graffiti, roof leaks, structural damage. Homeless NPCs may squat.
• Repair: Owner NPCs commission repairs. Player can affect through quests.
• Disaster recovery: After fire/war, damaged areas rebuild over time with NPC construction crews.

6.3 Infrastructure Systems

• Power: Generator/windmill/waterwheel networks. Power outages affect street lights (dark at night), forge temperatures, water pumps.
• Water: Aqueduct/well/pump systems. Water shortage affects NPC hygiene need satisfaction, crop growth, fire response capability.
• Sewage: Affects disease spread risk, NPC complaints, district desirability.
• Road maintenance: Potholes develop with traffic, affect vehicle speed. NPC work crews repair roads.

Gap 7: Voice-Controlled Combat AI & Formations (maya-combat-voice)#

What's Missing#

Phase 43 has behavior trees and pathfinding. Phase 79.1 has Euphoria-class ragdoll. But no phase covers tactical combat AI with voice control — the ability to command a squad like Rainbow Six, with NPCs that understand tactical concepts (breach, clear, cover, suppress, flank) and execute them with real military tactics.

Command Vocabulary#

MOVEMENT COMMANDS
├── "Move to [location]" — standard movement
├── "Stack up on [door/window]" — form breach line
├── "Hold position" — stay and defend current location
├── "Fall in" — return to formation near player
├── "Advance" — move forward maintaining formation
├── "Retreat to [location]" — fall back to position
├── "Flank left/right" — move to flanking position
├── "Go around back" — find rear entrance/approach

BREACH COMMANDS
├── "Breach and clear" — explosive/kick door, enter, clear room
├── "Flash and clear" — flashbang then enter
├── "Smoke and enter" — smoke grenade then enter
├── "Open and hold" — open door without entering, cover from outside
├── "Breach on my mark" — prepare breach, wait for "Go!" command

COMBAT COMMANDS
├── "Engage" / "Weapons free" — fire at will on hostiles
├── "Hold fire" / "Weapons hold" — cease fire, maintain position
├── "Suppress [target/area]" — sustained fire on area
├── "Take the shot" — sniper/marksman engage specific target
├── "Cease fire" — stop shooting immediately
├── "Cover me" — provide suppressive fire while player moves

FORMATION COMMANDS
├── "Wedge formation" — V-shape, balanced offense/defense
├── "Column" / "Single file" — line behind player
├── "Line abreast" — spread horizontal line
├── "Diamond" — 360° coverage formation
├── "Tight up" — close spacing for indoor
├── "Spread out" — wide spacing for open areas
├── "Bounding overwatch" — alternating advance with cover

POSTURE COMMANDS
├── "Go loud" — drop stealth, prioritize firepower
├── "Go quiet" / "Stealth" — suppress weapons, CQC preferred
├── "Aggressive" — push forward, overwhelm
├── "Defensive" — hold ground, conserve ammo
├── "Non-lethal" — tasers/batons only (for hostage situations)

Squad AI Architecture#

7.1 Autonomous Tactical Behavior (Between Commands)

When the player hasn't given a command, squad NPCs act autonomously:

• Threat assessment: Continuously evaluate nearby threats, assign engagement priority
• Cover seeking: Always maintain proximity to cover, peek/lean to engage
• Mutual support: Maintain firing lanes that don't cross friendly positions
• Ammo management: Call out low ammo, request resupply
• Casualty response: Drag wounded teammate to cover, apply first aid
• Noise discipline: In stealth mode, avoid unnecessary noise, whisper comms
• Callouts: "Contact! Two hostiles, ground floor, left side!" — automatic threat reporting with accurate spatial descriptors

7.2 Voice Command Priority System

Priority 5 (Maximum): Player voice command — always overrides
Priority 4: Self-preservation — under fire, take cover first
Priority 3: Team safety — rescue downed teammate
Priority 2: Standing order — last player command (e.g., "hold this position")
Priority 1: Autonomous tactical behavior
Priority 0: Idle/formation following

When a voice command arrives, it injects a Priority 5 behavior tree node that overrides current behavior. The NPC acknowledges verbally and executes. If execution is impossible (blocked path, no ammo), the NPC reports the problem verbally.

7.3 Multi-Agent Coordination

When the player gives a team command:

• Formation control: ORCA/RVO for collision avoidance, Boids for cohesion, named formation templates for structured positioning
• Breach synchronization: When "breach on my mark" is given, all stack members enter ready state. On "go!", synchronized entry with <100ms delta between first and last entry.
• Fire sector assignment: Each team member automatically assigns a sector to cover based on position and threat direction. No two members cover the same sector. Dynamic reassignment when teammates move or fall.
• Bounding overwatch execution: Team automatically splits into cover group and movement group, alternating advance. Player voice commands set pace and direction.

7.4 Personality-Driven Combat Behavior

Squad members aren't interchangeable — personality affects combat:

• High BRAVERY: Pushes aggressively, volunteers for point position, less likely to retreat
• High CAUTIOUSNESS: Takes extra time to clear corners, double-checks rooms, slower but more thorough
• High AGGRESSION: Prefers close engagement, may push too far ahead
• Low DISCIPLINE: Might fire before "weapons free," slower to cease fire
• VETERAN vs ROOKIE: Veterans peek corners smoothly, transition between positions fluidly. Rookies are jerky, take longer to set up, but improve over time.
• LOYALTY: High-loyalty NPCs prioritize player safety, will take bullets

Integration with Existing Systems#

Cross-Cutting Concerns#

Performance Budget#

All 7 gap systems must fit within a strict per-frame budget:

Technology Stack#

Suggested TODOS Phase Structure#

These 7 gaps could be organized as:

Phase 81: NPC Cognitive Core & On-Device SLM (Gap 1 + 4) — 140 tasks
  81.1 — Personality Neural Network (Tier 2 lightweight brain)
  81.2 — On-Device SLM Inference Engine (llama.cpp integration)
  81.3 — VRAM Budget Manager & CIG Scheduling
  81.4 — Teacher-Student Cognition Distillation
  81.5 — Unified Cognitive Loop (perceive→remember→think→act)
  81.6 — Attention System & Salience Scoring
  81.7 — Cloud LLM Fallback & Hybrid Reasoning
  81.8 — Cognitive LOD Manager (tier assignment, transitions, budgets)

Phase 82: Voice-as-First-Class Gameplay Input (Gap 2 + 5 + 7) — 145 tasks
  82.1 — On-Device ASR Pipeline (CarelessWhisper/Gemma integration)
  82.2 — Tactical Intent Classification (BERT-FID)
  82.3 — Natural Language Command Decomposition (SLM planner)
  82.4 — Spatial Reasoning from Voice (gaze + CLIP + entity DB)
  82.5 — Conversational NPC Interaction (voice dialogue mode)
  82.6 — Voice Negotiation & Persuasion System
  82.7 — Squad Tactical Command Execution
  82.8 — Multi-Agent Formation Control (ORCA + Boids)
  82.9 — Combat AI with Personality Variation & Voice Override Priority
  82.10 — TTS Feedback & Personality-Matched Voice
  82.11 — Wake Word & Activation Modes

Phase 83: NPC Emergent Agency & Living World Evolution (Gap 3 + 6) — 128 tasks
  83.1 — Autonomous Planning System (multi-horizon goals)
  83.2 — Reflection & Insight Generation
  83.3 — Emergent Goal Generation from Personality + Memory
  83.4 — Multi-NPC Emergent Cooperation
  83.5 — Procedural Building Interiors (shape grammar + furnishing)
  83.6 — Urban Evolution (construction, decay, gentrification)
  83.7 — Infrastructure Systems (power, water, sewage, roads)
  83.8 — NPC Lifecycle & Skill Progression (birth, aging, death, skills)
  83.9 — Voice-Driven Emergent Narrative Integration

What Makes This Beat GTA 6 and Everything Else#

The fundamental insight: Nobody has combined:

1. GTA 6's scale (10K+ NPCs via lightweight NNs)
2. Dwarf Fortress's depth (51+ personality facets, emergent social dynamics)
3. Stanford Generative Agents' memory and reflection
4. NVIDIA ACE's on-device SLM cognition
5. F.A.C.U.L.'s voice command architecture
6. SOCOM/Rainbow Six's tactical squad commands

...into a single coherent system. Each exists in isolation. The integration IS the innovation.

Dependencies#

Phase 43 (Neith AI Runtime) ─────────┐
Phase 40.10 (maya/souls) ────────────┤
Phase 78.1 (Voice ASR) ─────────────┤
Phase 79.1 (Euphoria ragdoll) ──────┤
Phase 79.11 (Crowd simulation) ─────┤
Phase 79.24 (NPC Life Simulation) ──┼──→ Phase 81 (Cognitive Core)
Phase 79.25 (NPC Relationships) ────┤          │
Phase 79.26 (NPC Occupations) ──────┤          ├──→ Phase 82 (Voice Gameplay)
Phase 79.27 (City Infrastructure) ──┤          │          │
Phase 79.28 (Economy) ──────────────┤          ├──→ Phase 83 (Emergent Agency)
Phase 79.29 (Justice System) ───────┤
Phase 79.31 (NPC-NPC Interaction) ──┤
Phase 79.32 (Simulation LOD) ───────┤
Phase 79.33 (World History) ────────┤
Phase 79.34 (Info Propagation) ─────┘

Note on Phase 79.11 (Crowd Simulation) integration: The cognitive LOD system (Phase 81.8) and the crowd simulation LOD (Phase 79.11) must share the same NPC entity pool. Tier 3/4 (statistical/schedule) NPCs from the cognitive system ARE the Tier 3/4 (Bulk/Impostor) crowd agents from Phase 79.11. When a crowd NPC promotes to Tier 2+ cognition, it simultaneously promotes in the crowd rendering LOD. The cognitive LOD manager (81.8) and the crowd LOD system (79.11) must use a shared priority scoring function to prevent conflicting tier assignments.

Note on multiplayer: These phases are designed for single-player and cooperative multiplayer. For MMO contexts (Phase 80), additional work is needed: NPC cognitive state must be server-authoritative, voice commands from multiple players to the same NPC need arbitration, and NPC memory of different players must be isolated. Phase 80 integration should be addressed as a follow-on phase.

Open Questions for Decision#

1. Primary on-device model: Gemma 4 E4B (Google, Apache 2.0, native audio input) vs Nemotron-Mini-4B (NVIDIA, optimized for role-play) vs fine-tuned Llama (Meta)? Recommendation: Gemma 4 for audio input capability, Nemotron for NPC dialogue quality. Support both.

2. ASR model: CarelessWhisper (streaming Whisper variant, proven) vs Gemma 4 E2B (native audio, smaller but newer)? Recommendation: CarelessWhisper for initial implementation (more mature streaming support), Gemma 4 as future option.

3. Voice command vs voice conversation routing: How does the system decide whether the player is giving a tactical command or talking to an NPC? Recommendation: Context-based — if the player is in a squad tactical scenario and looking at teammates, route to command pipeline. If looking at a civilian NPC, route to conversation. Player can override with push-to-talk modes.

4. VRAM allocation priority: When GPU memory is limited, should the system prioritize more Tier 1 NPCs (wider but shallower cognition) or fewer Tier 0 NPCs (deeper cognition for key NPCs)? Recommendation: Quality presets. "Immersion" preset favors breadth (many thinking NPCs). "Story" preset favors depth (fewer but deeply realized NPCs).

5. SLM fine-tuning strategy: Fine-tune one model per personality archetype (warrior, scholar, merchant) or one model with personality-conditioned prompts? Recommendation: Single model with personality prompts for flexibility. LoRA adapters for extreme specialization (e.g., unique companion characters).