Runbook: V1 DCC Bridge Host Loss (gd:bridge-host-loss)

Owner: delivery pipeline on-call (rota:delivery-pipeline). Last reviewed: 2026-08-14. Version: 1.

The integration that runs on somebody else's workstation. It takes the enrolled host with it, and the recovery is on the far side of a boundary the estate does not control, which is why its recovery time is the longest in the register — twenty-four hours, and that is a statement about a person's working day rather than about a restore.

This runbook is unusual and the difference is the point: almost none of the recovery is ours. dep:bridge-host rests on dep:dcc-bridge in the S10.5 substrate, and both live outside the estate's failure domain and outside its control. Nothing the operator runs restores them. What the estate owns is the gateway, the honest failure, and the decision about what happens to work that was in flight.

1. Detection and declaration#

Detection signal — an honest failure rather than a silence. libs/bellona/mcp-gateway/src/dcc-bridge-gateway.ts distinguishes the two ways this can go wrong, and the distinction is the diagnosis:

• No binding or transport registered → DccBridgeNotConfiguredError, thrown. That is a configuration fault on our side.
• Transport registered but not connected → a command result with status: 'failed' carrying an adapter.offline error. That is this scenario. It is never a fabricated success.

The contrast worth knowing under pressure: the estate also ships CloudAgentMvpSmokeGateway (cloud-agent-mvp-smoke.ts), a clearly-labelled in-memory simulation that fabricates a fixed objects: ['Camera','Cube','Key Light'] scene and always reports status: 'succeeded' with liveDeploymentExercised: false. If a bridge is "working" during this incident, check which gateway is bound before believing it.

Three capabilities stop: "import from a DCC session", "push a build to a host", "run a command on an enrolled host".

• Detection authority: delivery pipeline on-call, contact rota:delivery-pipeline.
• Declaration authority: production operations duty officer, contact rota:prod-ops.
• Gap, stated. docs/operations/on-call.md describes a weekly primary/secondary rotation and names no person and no rotation config; none of the register's rota: identifiers appears in it or in docs/operations/alert-routing.md. The name above is registered; the route from it to whoever is carrying the pager is not.

2. Communications#

All travel over dep:notification-channel, which this scenario leaves standing.

The third row is the one that matters. The person whose workstation this is performs the recovery. Every minute this runbook spends not contacting them is a minute of the twenty-four-hour objective spent on our side of a boundary we cannot cross.

3. Dependency assumptions#

• dep:identity is healthy. Re-enrolment is an authorization decision, and a bridge that re-enrols without one is a stranger's workstation running commands against the estate.
• dep:primary-store is healthy: it holds the record of what was imported, and it is what decides whether a re-run duplicates.

4. Recovery infrastructure#

Somebody else's workstation, and it is not a dependency in the S10.5 register and never will be. The estate provisions nothing here. Our side of the recovery is the gateway process, which is in place and healthy throughout — a bridge-host loss does not impair the gateway, it leaves it with nothing to talk to.

5. Secrets#

• The enrolment credential the bridge presents over the WebSocket command protocol. A re-enrolment issues a new one; the old one must not be reused, because a lost workstation is a lost credential until proven otherwise.

Recoverable out of band, and here that means something stronger than usual: the credential is reissued, not restored. There is no copy to recover.

6. Restore order#

1. dep:dcc-bridge — the bridge process comes back on the workstation and connects its transport.
2. dep:bridge-host — the enrolled host is only reachable once the bridge it rests on is connected, which is the substrate relation restated as a sequence.

No reference edge touches either, so the constraint is the substrate one. Doing it in the other order is not dangerous, it is impossible.

7. Validation#

• A round-trip command on the re-enrolled host returns a real bridge response. The gateway returns the bridge's own output as the command result and never a fabricated scene, so a genuine response is evidence and a plausible one is not.
• The result does not carry adapter.offline, and the transport is not the simulation: check for transport: 'mcp-in-memory-cloud-agent-simulation' and liveDeploymentExercised: false, which together mean the smoke gateway answered and the workstation did not.
• One import completes end to end and lands in the relational store.

8. Fencing — NOT APPLICABLE, with a real reason#

Nothing is promoted. But there is a genuine two-writers question here and it is worth naming so that its absence from this section is a decision: a workstation that comes back while a replacement is enrolled is two bridges claiming the same host. The estate has no fencing token to reject the older one — there is no fencing anywhere in this estate — so the control is the enrolment credential in §5: revoke the old one before issuing the new.

That is a human control and it is the weakest step in this runbook, exactly as §8 of v1-disaster-recovery.md is in that one.

9. Failover#

There is nothing to fail over to. A second workstation is a second enrolment performed by a second person, which is a recovery rather than a failover, and it is the same procedure as this one with a different host.

10. Reconciliation#

The question is whether an import that was in flight at the loss landed. The gateway's failure is honest — status: 'failed' with adapter.offline — so a command that was dispatched and not answered is distinguishable from one that was never dispatched, provided the dispatch was recorded.

Re-run only imports whose result is absent, not those whose result is a recorded failure: the second class was answered.

11. Failback#

There is no alternate topology to return from; the estate stayed where it was. Failback is the point at which the re-enrolled host has completed one real round-trip and one import, and the old enrolment credential has been revoked. Record the instant — recovery time runs from the fault.

12. What this runbook does not cover#

• Anything on the workstation. The estate does not administer it, cannot restore it, and this runbook will not describe steps somebody else performs on hardware we cannot see.
• The simulation gateway. cloud-agent-mvp-smoke.ts is a labelled test double; if it is bound in production that is a configuration incident, not this one.
• Engine-specific bridge failures (blender, godot, houdini, davinci, maya, 3dsmax, unreal). The gateway takes no compile dependency on any of them and neither does this procedure.

13. Machine-checkable statement#

{
  "id": "runbook:v1-bridge-host-loss",
  "scenarioId": "gd:bridge-host-loss",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "delivery pipeline on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    },
    {
      "audience": "provider",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 3600000
    }
  ],
  "assumes": ["dep:identity", "dep:primary-store"],
  "recoveryInfrastructureDependencyId": "a workstation the estate does not administer",
  "secrets": [
    {
      "id": "bridge enrolment credential",
      "heldInDependencyId": "dep:dcc-bridge",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:dcc-bridge", "dep:bridge-host"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "a round-trip command returns the bridge's real response rather than adapter.offline",
    "the transport is not mcp-in-memory-cloud-agent-simulation and liveDeploymentExercised is not false",
    "one import completes end to end and lands in the relational store"
  ],
  "failback": "one real round-trip, one completed import, and the old enrolment credential revoked; the estate never left its topology"
}