# Danu Server Meshing & Nephthys Persistence

Mawu's promise — a roleplay realm that holds a thousand inhabitants in one
seamless world, hosted by a community operator the platform does not trust — has
exactly two load-bearing systems underneath it. **Danu** is the meshing control
plane: it makes many simulation nodes behave as one world by holding _single
authoritative writer per entity_, handing authority across partition boundaries
atomically, and bending under load instead of breaking. **Nephthys** is the
decoupled source of truth: an event-sourced, snapshot-checkpointed store that
brokers read/write views between mesh nodes and rehydrates a replacement node
after a crash with nothing lost. The two are designed as a pair — Danu owns _who
simulates what, right now_; Nephthys owns _what is true, durably_ — so a sim
node can die, be preempted off a Spot instance, or be split in two without a
player ever reconnecting. This is the architecture that lets persistent RP state
ride cheap, preemptible compute: the truth is never in the node.

Both follow named industry lessons rather than invention: the Star Citizen
replication-layer split (authority decoupled from the durable layer), the
SpatialOS failure (never shard tightly-coupled physics across workers), and the
EVE Online time-dilation valve (slow the clock, never drop the player). This
page is the deep dive behind the "Realm Backbone" group of the orientation hub
[../V7_ARCHITECTURE.md](../V7_ARCHITECTURE.md). It is the
persistence-and-meshing companion to
[./moremi-realm-server-and-netcode.md](./moremi-realm-server-and-netcode.md)
(the authoritative sim loop and the client↔realm wire), draws its character and
economy aggregates from
[./nana-data-model-and-pheme-voice.md](./nana-data-model-and-pheme-voice.md),
and feeds the orchestration story in
[./hosting-fleet-data-and-compliance.md](./hosting-fleet-data-and-compliance.md).

## What ships, honestly

The meshing and replication _logic_ is real, deterministic, and guarded by
falsifiable CI gates — but it is library-grade simulation, not a running
distributed cluster, and the daemons that wrap it are health endpoints. Four
honest layers:

- **Real and exercised by tests.** Danu's two-phase atomic authority handoff
  over a Nephthys-modelled ledger, grid interest management with a bounded
  working set, co-location pinning, density-driven split/merge, and the
  time-dilation overload valve are all genuine algorithms with named numeric
  budgets, covered by 22 `#[test]` cases in
  `apps/v7/danu-mesh-cluster/src/lib.rs`. Nephthys's event-sourced store,
  single-writer replication broker with fencing tokens, deterministic crash
  rehydrate, and byte-verified persistence API are real, covered by 15 tests in
  `apps/v7/nephthys-replica-service/src/lib.rs`. Each eval emits a typed report
  with a strict `passed()` predicate that would fail against a regression.
- **The eval harnesses are deterministic in-process models, not live multi-node
  systems.** A "node death" is a boolean flip on an in-process struct
  (`source_node.alive = false`, `lib.rs:1855`); the mesh "cluster" is a
  two-partition `BTreeMap`. Their value is as golden CI gates that pin exact
  invariants (zero lost entities, zero duplicate authority, zero reconnects over
  two million crossings) — not as a deployed control plane.
- **The daemons are health-only.** Both binaries' `run_service()` bind a TCP
  health server that emits the crate's `ServiceDescriptor` JSON
  (`"status":"ready"` plus the capability list, `service_contract.rs`); each
  `main.rs` is one line. There is no accept loop ingesting real sim-node
  traffic, no live replication socket. The Danu daemon listens on `:47202`,
  Nephthys on `:47203`, and both serve only `/health`.
- **Persistence is an in-memory event-sourced store with a documented Postgres
  target.** `NephthysEventSourcedStateStore` (`lib.rs:1271`) is a `Vec` of
  events plus a `BTreeMap` of snapshots — its byte-canonical encoding and
  deterministic hashing are precisely what make the monolith's Postgres+pgvector
  durable backing a safe swap, but that backing is not in this crate. The one
  place V7 touches a _reused_ persistence substrate today is the identity path:
  `libs/v7/substrate-bridge` consumes the real V6 Egbe Ori event store,
  projection materializer, and operator-read audit primitives.

## The three-part backbone

Danu, Nephthys, and the realm wire are three separable concerns that compose
into one guarantee. The realm-protocol crate (`libs/v7/realm-protocol`) carries
the bytes; Danu decides which node owns an entity and routes interest; Nephthys
is the authority broker and the durable log. The sim nodes themselves are the
Moremi realm servers (`apps/v7/moremi-realm-server`), which depend on the Danu
crate directly (`DanuMeshCluster`, `DanuMeshNodeId`) and consume Nephthys
through a signed-receipt abstraction.

```mermaid
flowchart TB
    subgraph nodes["Moremi sim nodes (Rust, authoritative)"]
      W["node:west<br/>writer for cell range"]
      E["node:east<br/>writer for cell range"]
    end
    subgraph danu["Danu mesh control plane (:47202)"]
      AOI["grid interest mgmt<br/>≤64 working set / ≤4096 B frame"]
      HO["2-phase handoff<br/>Freeze → Transfer → Ack → Release"]
      SM["density split/merge<br/>+ time-dilation valve"]
    end
    subgraph neph["Nephthys (:47203)"]
      BRK["replication broker<br/>fencing-token leases<br/>read-only views"]
      ES["event-sourced store<br/>snapshot every 3 events<br/>realm ∥ character aggregates"]
      PROJ["independent projections<br/>+ residency tags"]
    end
    W -- "intent / write (lease + token)" --> BRK
    E -- "read-only view" --> BRK
    HO -- "freeze / transfer / commit owner" --> BRK
    BRK --> ES --> PROJ
    ES -- "rehydrate (snapshot + tail replay)" --> W
    AOI -- "interest set" --> nodes
    SM --> HO
```

## Danu — meshing for a 1000+ population

Danu's contract (`arch§"Danu — Server Meshing"`) is six invariants. The crate
implements each as a data structure plus a deterministic eval that proves it.

### Single-writer authority and the atomic two-phase handoff

Exactly one node holds write authority over an entity; everyone else reads a
view sourced from Nephthys. Crossing a partition boundary therefore means
_moving authority_, and the dangerous window is two writers (or zero) for one
entity. `DanuMeshCluster::move_entity_across_boundary` (`lib.rs:1822`) runs the
four-record sequence against a `DanuNephthysHandoffLedger`: it **freezes** the
entity (`frozen = true`), records `Freeze` then `Transfer` (`lib.rs:1850`),
**commits** the new owner into the ledger, records `Ack`, flips the entity's
`owner_node_id`, and finally records `Release` and unfreezes (`lib.rs:1874`).
The mid-handoff crash case is explicit: when the eval kills the source node
between Transfer and Ack, recovery is modelled (`alive` toggles back, the
recovered counter increments) so the handoff still completes against Nephthys.

The gate is `run_danu_mesh_handoff_eval` (`lib.rs:1973`): **200 entities cross a
boundary 10,000 times each — two million handoffs** — with a deliberate
mid-handoff node death injected at crossing 5,000.
`DanuMeshHandoffEvalReport::passed` (`lib.rs:1086`) demands
`total_handoffs == expected_handoffs`, every one of the four phase counters
equal to that total, `mid_handoff_node_deaths > 0` _and_ fully recovered,
`connected_clients_before == connected_clients_after`, `client_reconnects == 0`,
and empty `lost_entities` / `duplicate_authority_entities` /
`owner_mismatch_entities`. The owner-mismatch check cross-references the spatial
partition and the ledger, so an entity whose position and ledger owner disagree
fails the gate — the single-writer invariant is enforced, not asserted.

### Interest management: the bounded working set

A client never receives the whole realm. `run_danu_area_of_interest_eval`
(`lib.rs:2002`) populates a **1,000-inhabitant** grid (40×25 cells, `DANU_AOI_*`
constants) and, for every inhabitant, computes a working set that is culled
three ways: by **proximity** (radius 2 cells), by **line-of-sight**, and by
**channel** (Public / Crew / Trade subscriptions must intersect). The report's
`passed()` (`lib.rs:1167`) requires that the maximum working set stay `≤ 64`
entities and the maximum per-frame estimate stay `≤ 4096` bytes, that all three
culling paths actually removed entities (each total `> 0`), and that no working
set is empty. Because the budget is a bound on _visible_ density rather than
population, per-client bandwidth is independent of realm size — the same
property the realm-protocol delta layer enforces with its 32-snapshot
ack-relative ring and 64–256 kbit/s budget (`REALM_DELTA_SNAPSHOT_RING_SIZE`,
`realm-protocol/src/lib.rs:58`), which is the wire Danu's interest sets feed.

### Co-location and density-driven split/merge

The SpatialOS lesson is that tightly-coupled physics must never straddle nodes.
`run_danu_colocation_eval` (`lib.rs:2073`) builds **128 coupled pairs** —
`PhysicsJoint`, `VehicleRider`, `CombatGrapple` — and pins both members of each
pair to one anchor node; the gate fails on any `split_pair_ids`, any missing
assignment, or any pair whose interaction frequency drops below the 60 Hz
co-location threshold. Splitting is the hard part, so Danu proves it _static
first_: `run_danu_dynamic_meshing_eval` (`lib.rs:2121`) begins by running the
full static handoff eval and refusing to proceed unless it passed, then drives a
crowd to gather (density crosses the split threshold of 120) and disperse (below
the merge threshold of 40). The smoothness budget caps remaps at 16 entities per
tick and per-tick cost at one 60 Hz frame (16,667 µs), and the topology must
return to `Static` with the original partition count — a split that never merges
back fails the gate.

### The overload valve: time dilation, not authority drop

When a node cannot simulate a hot region in real time, dropping authority or
disconnecting players is the worst outcome. Danu instead slows the region's
clock, EVE-style. `danu_overload_clock_percent` (`lib.rs:2500`) is the real
formula: at or below capacity the clock runs at 100%; above it, the clock is
`capacity × 100 / load`, clamped to a **10% floor**. `run_danu_overload_eval`
(`lib.rs:2237`) drives 256 players through 36 ticks with a load spike to 18,000
units against a 1,000-unit capacity, and the gate (`DanuOverloadEvalReport`)
requires that the clock _did_ dilate, never fell below the floor, dropped
**zero** players, dropped **zero** authority records, preserved event ordering,
and kept per-player event lag at zero — fairness under dilation, not just
survival.

### Degradation: crash, capacity, control-plane loss

`run_danu_degradation_eval` (`lib.rs:2345`) is the chaos gate, composing three
scenarios. **Node failure** must rehydrate from Nephthys (the
`RehydratedFromNephthys` event, a present `fallback`, and `playable_after`).
**Capacity exhaustion** must _queue_ the 64 excess joins rather than reject them
(`JoinQueued`). **Control-plane loss** must fall back to single-node operation
(`SingleNodeFallbackEnabled`). Across all three, the aggregate gate demands zero
client reconnects, zero lost entities, and zero authority drops — the realm
degrades, it does not break. This scenario is also where the Danu↔Nephthys seam
is most visible: the Spot-reclaim path (`handle_spot_preemption`, `lib.rs:517`)
shuts down a preempted game server and reallocates a replacement that restores
from a `DanuNephthysSessionCheckpoint` with no player-visible state loss — the
literal reason persistent RP realms are safe on preemptible compute. The Agones
fleet, autoscaler, and cost-tier machinery that surround this live in the Danu
crate too but belong to
[./hosting-fleet-data-and-compliance.md](./hosting-fleet-data-and-compliance.md).

## Nephthys — persistence and the replication layer

Nephthys is the decoupled truth. Its two jobs are to be the durable
event-sourced store and to be the replication broker that enforces the
single-writer rule Danu depends on.

### Event-sourced aggregates and deterministic rehydrate

State is an append-only event log with periodic snapshot projections.
`NephthysEventSourcedStateStore::append_event` (`lib.rs:1296`) assigns a
monotonic sequence, hashes the event, and **checkpoints the aggregate every
third version** (`NEPHTHYS_CHECKPOINT_INTERVAL = 3`). Critically, realm state
and character state are **separate aggregates** (`NephthysAggregateKind::Realm`
vs `Character`) with independent projections, so a character can travel a
federation corridor carrying only its own aggregate. Recovery is snapshot + tail
replay: `rehydrate_aggregate` (`lib.rs:1357`) loads the latest snapshot and
replays only events after its `through_sequence`. The crash gate
`run_nephthys_state_store_rehydrate_smoke` (`lib.rs:1525`) builds a realm with
two characters, rehydrates a _primary_ and a _replacement_ node, and diffs them
with `diff_nephthys_rehydrated_nodes` (`lib.rs:1471`); `passed()` (`lib.rs:766`)
requires that snapshots and tail replay were both used, that the ledger diff is
empty, that `projection_hash_before == projection_hash_after`, and that the
replacement reaches the ledger head. A replacement node is bit-identical to the
node it replaces — that is the whole point.

### The single-writer replication broker (fencing tokens)

`NephthysReplicationBroker` (`lib.rs:878`) is where the single-writer invariant
becomes a _write protocol_. `grant_authority` (`lib.rs:903`) issues an
incrementing **fencing token** with a tick-bounded lease; `read_only_view`
(`lib.rs:923`) gives a non-owner a rehydrated, sequence-stamped view; and
`submit_write` (`lib.rs:942`) admits a mutation only if the presenting node owns
the lease, presents the _current_ fencing token, and writes before the lease
expires — otherwise it returns a precise `NephthysSingleWriterError`
(`LeaseOwnedByDifferentNode`, `StaleFencingToken`, `ExpiredAuthorityLease`,
`ReadOnlyReplicaWrite`). `run_nephthys_single_writer_eval` (`lib.rs:1581`)
exercises two legitimate owners, three hostile writes (a read-only replica, a
stale-token grant, an off-lease write), and an authority _transfer_ (a new lease
with a fresh token that fences the old owner's stale token). The gate
(`lib.rs:866`) demands `≥ 2` accepted writes, `≥ 3` blocked writes, non-empty
read-only views, and — the sharp invariant — that the ledger grew by **exactly**
the accepted count, so a blocked write can never have leaked a side effect.

### Byte-verified persistence and hosting-tier migrations

`NephthysPersistenceApi` (`lib.rs:1061`) is the realm-facing surface — bootstrap
a realm, mutate world objects, post a double-entry economy line, create a
character, grant inventory, set property/vehicle/position. The smoke gate
`run_nephthys_persistence_api_smoke` (`lib.rs:1746`) writes a full character,
captures `canonical_bytes()`, restarts the writer onto a _replacement_ node, and
proves the post-restart bytes are exactly equal
(`NephthysByteVerification.exact_match`) for both the character and realm
aggregates — plus that residency tags survive on every property, vehicle, and
ledger line. Two further evals prove the hosting-tier story end to end:
`run_listen_host_migration_eval` (`lib.rs:2505`) migrates a peer-hosted Listen
session to a successor host, using a fencing token to reject the stale host's
write while byte-verifying zero character-state loss; and
`run_realm_promotion_eval` (`lib.rs:2685`) promotes a Solo-authored realm to
Dedicated then Meshed with a _byte-identical_ composition lock file at every
tier, migrating persistence into Nephthys at the Solo→non-Solo boundary
(byte-verified). A realm grows from one author to a thousand inhabitants without
a rebuild and without losing a save.

## The trust seam Danu and Nephthys ride on

Meshing only matters if a hostile operator cannot forge the writes it routes.
Two real seams enforce that. On the wire, `libs/v7/realm-protocol` binds every
cross-trust message to a platform-issued `RealmSecurityToken` (HMAC-SHA256 over
the claims) and a sequence-numbered `RealmSignedIntent`; its
`RealmEventTamperEvalReport::passed` (`realm-protocol/src/lib.rs:1131`) proves
that only a valid signed client intent mutates authoritative state and that the
post-hostile state hash is _identical_ to the baseline — validation strictly
precedes mutation. On the identity side, `libs/v7/substrate-bridge` is the one
place V7 wires a reused persistence substrate: `V7IdentityFirewall` projects a
platform principal into an opaque per-realm handle, and the integrated facade
writes a Nàná character memory through the **real V6 Egbe Ori event store**
(`PartitionedPostgresOriEventStore`, `OriProjectionMaterializer`,
`OperatorReadAuditStore`) and reads it back through the audited operator-read
path — proving the realm never sees a platform account id. Moremi itself
consumes Nephthys not as a network service but as a signed-receipt abstraction:
`MoremiNephthysMutationReceipt` is HMAC-signed and
`validate_moremi_nephthys_receipt` rejects realm-mismatched, actor-mismatched,
rule-mismatched, replayed, or unsigned authority mutations before they touch
state.

## Where it runs (and doesn't yet)

The accurate mental model: **Danu and Nephthys are a tested decision-and-truth
core, not a deployed cluster.** The handoff sequencer, the fencing-token broker,
the rehydrate path, and the byte-verified API are production-grade Rust with
golden gates; the daemons around them are health endpoints, and the evals are
deterministic single-process fixtures. To become the runtime the monolith
describes, three wires remain: bind the Nephthys store to durable
Postgres+pgvector (the byte-canonical design already makes this a swap, not a
rewrite), bind the Danu control plane to real sim-node membership and the Agones
allocation path, and run the realm-protocol frames over the live V6 Egbe
gateway. None of that is faked here — it is honestly absent, with the seams
shaped to accept it.

## Related

- [../V7_ARCHITECTURE.md](../V7_ARCHITECTURE.md) — the orientation hub and the
  full trust-boundary, determinism, and reuse model
- [./moremi-realm-server-and-netcode.md](./moremi-realm-server-and-netcode.md) —
  the authoritative sim loop, prediction/reconciliation, and the realm wire that
  Danu's interest sets and Nephthys's writes flow through
- [./nana-data-model-and-pheme-voice.md](./nana-data-model-and-pheme-voice.md) —
  the character and economy aggregates Nephthys persists, and the proximity
  voice budget that gates the same 1000+ population target
- [./hosting-fleet-data-and-compliance.md](./hosting-fleet-data-and-compliance.md)
  — the Agones fleet, cost tiering, Spot reclaim, and residency machinery that
  surround the meshing and persistence cores