Eve library

ADR-0007: QUIC / WebTransport for Multiplexed Control and Media Signaling

ADR

Revision 1

On this page
  • Status: Accepted (decision: defer, with a staged adoption trigger)
  • Date: 2026-06-21
  • Authors: Bellona Remote Control Working Group, Claude
  • Reviewers: Bellona, Oshun Platform, Oshun Security
  • Task: 180.C.25.11

Context and Problem Statement#

The Bellona remote-control transport (180.C.25.01–180.C.25.06) carries every host↔gateway interaction over a single host-initiated outbound WebSocket Secure (WSS) tunnel on TCP/443 (180.C.25.03). On top of that one byte pipe we built, in application code:

  • an app-level multiplexer of seven logical channels — control, events, logs, file_transfer, artifact_metadata, stream_signaling, emergency_stop — each with its own ordering/reliability semantics (180.C.25.05, apps/bellona/remote-host/src/channel-multiplexer.ts);
  • a priority scheduler + backpressure controller over eight traffic classes (180.C.25.06, apps/bellona/remote-host/src/backpressure-scheduler.ts);
  • session resume tokens so a dropped tunnel reconnects without re-pairing (180.C.25.03, libs/bellona/remote-protocol/src/transport.ts).

This question (180.C.25.11) is whether to adopt QUIC (RFC 9000) and its browser-facing API WebTransport (W3C) for future multiplexed control and media signaling, defer it, or reject it — and, if we might adopt later, to record the migration requirements now while the design is fresh.

This ADR is a forward-looking transport-substrate decision. It does not change 180.C.25.04's WebRTC/TURN decision for the media plane (pixels/audio): that already uses SRTP over ICE/DTLS and is out of scope here except where stream signaling rides the control tunnel.

Decision Drivers#

  • Head-of-line (HOL) blocking. Our seven channels share one TCP connection. TCP guarantees a single in-order byte stream, so a lost segment belonging to a large file_transfer stalls the kernel's delivery of every later byte — including a queued control or events frame — until the retransmit lands. Our app-level mux cannot fix this: it only sees bytes after TCP has reassembled them in order. (Our scheduler does prevent HOL blocking on the send side, and our mux prevents it on the receive demux side once bytes arrive — but the TCP socket itself remains a single ordered stream beneath both. The emergency_stop guarantee today rests on the scheduler sending it first and the socket being healthy; under packet loss the kernel can still delay its bytes behind an earlier lost segment.)
  • Native multiplexing. QUIC carries many independent streams over one connection; loss on one stream does not block others. This is the protocol- level version of exactly what our app-level mux approximates.
  • Connection migration. QUIC connections are identified by a Connection ID, not the 4-tuple, so a NAT rebinding or Wi-Fi→cellular handoff survives without a new handshake. Our tunnel handles this today with reconnect + resume tokens (180.C.25.07 NAT-change test); QUIC would make it seamless.
  • 0-RTT resume. QUIC/TLS 1.3 can resume a session and send application data in the first flight (0-RTT). This overlaps with our resume tokens (180.C.25.03) but at the transport layer.
  • Firewall/NAT traversal on 443. Our hard requirement is reachability from a studio/home/coffee-shop network with no inbound ports. WSS on TCP/443 is essentially always allowed. QUIC is UDP/443, which a non-trivial fraction of corporate/hotel/guest networks block or throttle. Any QUIC adoption must keep a TCP fallback or we lose reachability — the entire point of the tunnel.
  • Browser support for the Control Room. WebSocket is universally supported. WebTransport shipped in Chromium-based browsers and is available in recent Firefox; Safari/WebKit support is still limited/experimental. The Control Room is browser-first and must support Safari operators, so WebTransport cannot be the only control transport without excluding Safari.
  • Operational maturity & cost. WSS terminates behind any standard L7 load balancer/reverse proxy (nginx, ALB, Cloudflare) with mature tooling. HTTP/3 termination is supported by these but is younger; UDP load balancing, MTU/PMTU discovery, and amplification-attack mitigation are additional operational surface for a small team.
  • Don't fork the protocol. We have one protocol package and a working app-level mux/scheduler. A second transport must reuse the same RemoteCommandEnvelope, channel descriptors, and traffic classes, not fork them.

Considered Options#

Option 1: Adopt QUIC/WebTransport now as the primary control transport#

Replace the WSS tunnel with a WebTransport (browser) / raw-QUIC (host) connection; map our seven channels onto native QUIC streams (one stream per channel, or a stream per bulk object on file_transfer); use a QUIC datagram for emergency_stop so it is never blocked by any stream.

Rejected for now. Three blockers:

  1. Reachability regression. UDP/443 is blocked on enough networks that making it primary would strand operators the WSS tunnel reaches today. We would have to ship both transports anyway, so QUIC-primary buys complexity without removing the WSS path.
  2. Safari Control Room. WebTransport is not dependable on Safari/WebKit yet. A browser-first Control Room that must run on Safari cannot drop the WebSocket control path.
  3. Operational immaturity for our scale. The team is shipping an MVP; adding UDP load balancing, HTTP/3 termination, and amplification mitigation now is cost we cannot justify before the control plane has real load.

Option 2: Reject QUIC/WebTransport permanently#

Commit to WSS-over-443 + the app-level mux forever; never revisit.

Rejected. The HOL-blocking limitation is real and structural: as we add high-throughput file_transfer and richer events/stream_signaling traffic to the shared TCP connection, a single lost segment in a bulk transfer measurably delays control and event frames in a way no application code can remove. QUIC is the correct long-term substrate for multiplexed control + media signaling, and the migration cost is bounded precisely because we already modelled channels and traffic classes as first-class. Closing the door now would be a mistake.

Option 3: Defer, behind an explicit adoption trigger, keeping the design QUIC-ready (chosen)#

Keep WSS-over-443 as the shipping transport. Treat QUIC/WebTransport as a planned second transport that we add when a measured trigger fires, not on a calendar date. Keep the protocol and the mux/scheduler transport-agnostic so the swap is a transport adapter, not a protocol rewrite.

This option is chosen.

Decision Outcome#

Defer adoption. Ship and operate the WSS-over-443 tunnel with the app-level multiplexer (180.C.25.05) and priority/backpressure scheduler (180.C.25.06). Do not build QUIC/WebTransport for the MVP.

Adopt QUIC/WebTransport as a second, negotiated control transport (never a silent replacement) only when any of these triggers is met and recorded with evidence:

  • T1 — measured HOL pain. Production telemetry shows control/event-frame delivery latency regressing under concurrent file_transfer load on the shared TCP connection, beyond the 180.C.25 performance budget, attributable to TCP-level HOL blocking (i.e. the scheduler already sent the frame but the socket delayed its bytes).
  • T2 — mobility pain. Reconnect/resume churn from NAT rebinding or network handoff (180.C.25.07 NAT-change class) becomes a top operator complaint that QUIC connection migration would remove.
  • T3 — browser readiness. WebTransport reaches dependable support across the Control Room's required browser matrix including Safari/WebKit, removing the need to keep WebSocket as a parallel browser control path.

Adoption is additive and negotiated: the host advertises transport support in its handshake/discovery TXT (180.C.25.01), the gateway picks the best mutually supported transport, and WSS-over-443 remains the guaranteed fallback for networks that block UDP/443 and for Safari. We will never have a configuration that cannot fall back to the TCP tunnel.

Why defer rather than adopt or reject#

The benefit (removing TCP HOL blocking, seamless migration, 0-RTT) is real but not yet felt at MVP scale; the cost (dual transport, UDP reachability gaps, Safari gap, HTTP/3 ops) is immediate. Deferring captures the upside option while paying none of the cost now, and the "keep it QUIC-ready" rules below make the later swap cheap.

Keeping the Design QUIC-Ready (no-regret work to do now)#

These cost little today and make a later QUIC adapter a drop-in. The current code already satisfies most of them:

  • Transport seam already abstracts the socket. RemoteTunnelTransport (apps/bellona/remote-host/src/gateway-tunnel.ts) is a minimal connect/send/close/onMessage/onClose interface. A WebTransport/QUIC adapter implements the same interface; the tunnel state machine, mux, and scheduler are unchanged. Keep this seam narrow — do not leak WebSocket-specific types into the tunnel/mux/scheduler.
  • Channels and traffic classes are protocol data, not transport data. RemoteTransportChannel, the channel descriptors, and the eight traffic classes live in libs/bellona/remote-protocol/src/channel.ts. A QUIC mapping reuses them verbatim: each channel → a QUIC stream; emergency_stop (our only unordered, neverBlocked channel) → a QUIC datagram; file_transfer bulk objects → one stream per object so loss is isolated per transfer.
  • Resume tokens are transport-agnostic. RemoteTunnelResumeToken (libs/bellona/remote-protocol/src/transport.ts) is independent of how the bytes move; it maps cleanly onto QUIC connection migration + TLS 1.3 session resumption when we adopt.
  • Frame envelope already carries channel + sequence. RemoteChannelFrame tags channel/sequence/traffic-class explicitly, so over QUIC the per-stream ordering is native and our sequence becomes a redundant integrity check rather than the ordering mechanism — no schema change required.

Migration Requirements (if/when a trigger fires)#

Recorded now so adoption is execution, not rediscovery. Each becomes its own 180.C.25.11.x task at adoption time:

  1. Transport negotiation. Extend the host handshake (180.C.25.03) and mDNS TXT (180.C.25.01) with a supportedTransports list (e.g. ["wss-443","webtransport-h3-443"]). The gateway selects the best mutually supported transport and records the choice in session metadata + audit. WSS must always be in the list.
  2. QUIC/WebTransport adapter. Implement RemoteTunnelTransport over WebTransport (Control Room browser) and over a raw-QUIC library (host service, e.g. a quinn/msquic-backed Node/Rust binding). No changes to the tunnel state machine, mux, or scheduler.
  3. Channel→stream mapping. Map each RemoteTransportChannel to a dedicated bidirectional QUIC stream; map emergency_stop to a QUIC datagram (or an urgent dedicated stream) so it is delivered without any stream's HOL blocking; map each file_transfer bulk object to its own unidirectional stream so loss is isolated per object. Re-run the 180.C.25.05 HOL proof against the QUIC adapter.
  4. Backpressure mapping. Replace the socket-buffer bytesQueued() backpressure signal (RemoteSendChannel) with QUIC per-stream flow-control credit + connection-level flow control. The selectNextToSend priority logic and shed policy are unchanged; only the canSend() source changes per stream.
  5. 0-RTT vs replay safety. If 0-RTT is enabled, gate it to idempotent / replay-safe traffic only. emergency_stop and approval must remain safe under 0-RTT replay; non-idempotent engine_command/file_transfer initiation must wait for 1-RTT confirmation. Reuse the existing idempotency-key dedupe window (180.C.25.03) as the anti-replay backstop.
  6. Infrastructure / deployment. Add UDP/443 ingress + HTTP/3 termination to the Hetzner (180.C.25.08) and AWS (180.C.25.09) reference deployments: open UDP/443, configure HTTP/3 on the LB/proxy, set QUIC connection-ID-aware UDP load balancing, handle PMTU/MTU and the QUIC anti-amplification limit, and keep TCP/443 WSS listening in parallel. Add Alt-Svc advertisement so capable browsers upgrade.
  7. Fallback + reachability probe. On connect, attempt QUIC and fall back to WSS on UDP-block/timeout within a bounded budget, recording which transport won per session for diagnostics (180.C.25.12). Never strand a session that can only reach TCP/443.
  8. Security review. Re-run the threat model (ADR-0002) for the QUIC path: amplification, connection-ID linkability/migration privacy, 0-RTT replay, datagram unreliability for emergency_stop (must have an ack/retry at the app layer since datagrams are unreliable), and parity of the gateway-trust / E2E-encryption model (ADR-0003) across both transports.
  9. Impairment test parity. Extend the 180.C.25.07 impairment suite (apps/bellona/remote-host/src/network-impairment.ts) with a QUIC-adapter transport so every impairment (latency, loss, gateway restart, sleep/wake, NAT change, Wi-Fi disconnect, lid close) is asserted to recover over QUIC too — especially NAT change, which should now succeed via connection migration without a reconnect.

Consequences#

  • The MVP ships on WSS-over-443 with the app-level mux + scheduler; no QUIC work is scheduled for the MVP.
  • The transport seam, channel descriptors, traffic classes, frame envelope, and resume tokens stay transport-agnostic so the later swap is an adapter, not a rewrite. Code review must reject any WebSocket-specific type leaking past the RemoteTunnelTransport seam.
  • When a trigger fires, adoption is additive and negotiated, WSS remains the guaranteed fallback (UDP-blocked networks, Safari), and the migration- requirements list above becomes the 180.C.25.11.x work breakdown.
  • The emergency_stop guarantee is explicitly carried forward: over QUIC it must use an app-layer ack/retry because QUIC datagrams are unreliable — this is called out in migration requirement 8 so the kill switch is never weakened by the move.