- Status: Accepted (decision: defer, with a staged adoption trigger)
- Date: 2026-06-21
- Authors: Bellona Remote Control Working Group, Claude
- Reviewers: Bellona, Oshun Platform, Oshun Security
- Task:
180.C.25.11
Context and Problem Statement#
The Bellona remote-control transport (180.C.25.01–180.C.25.06) carries every
host↔gateway interaction over a single host-initiated outbound WebSocket Secure
(WSS) tunnel on TCP/443 (180.C.25.03). On top of that one byte pipe we built,
in application code:
- an app-level multiplexer of seven logical channels —
control,events,logs,file_transfer,artifact_metadata,stream_signaling,emergency_stop— each with its own ordering/reliability semantics (180.C.25.05,apps/bellona/remote-host/src/channel-multiplexer.ts); - a priority scheduler + backpressure controller over eight traffic classes
(
180.C.25.06,apps/bellona/remote-host/src/backpressure-scheduler.ts); - session resume tokens so a dropped tunnel reconnects without re-pairing
(
180.C.25.03,libs/bellona/remote-protocol/src/transport.ts).
This question (180.C.25.11) is whether to adopt QUIC (RFC 9000) and its
browser-facing API WebTransport (W3C) for future multiplexed control and
media signaling, defer it, or reject it — and, if we might adopt later,
to record the migration requirements now while the design is fresh.
This ADR is a forward-looking transport-substrate decision. It does not
change 180.C.25.04's WebRTC/TURN decision for the media plane (pixels/audio):
that already uses SRTP over ICE/DTLS and is out of scope here except where stream
signaling rides the control tunnel.
Decision Drivers#
- Head-of-line (HOL) blocking. Our seven channels share one TCP connection.
TCP guarantees a single in-order byte stream, so a lost segment belonging to a
large
file_transferstalls the kernel's delivery of every later byte — including a queuedcontroloreventsframe — until the retransmit lands. Our app-level mux cannot fix this: it only sees bytes after TCP has reassembled them in order. (Our scheduler does prevent HOL blocking on the send side, and our mux prevents it on the receive demux side once bytes arrive — but the TCP socket itself remains a single ordered stream beneath both. Theemergency_stopguarantee today rests on the scheduler sending it first and the socket being healthy; under packet loss the kernel can still delay its bytes behind an earlier lost segment.) - Native multiplexing. QUIC carries many independent streams over one connection; loss on one stream does not block others. This is the protocol- level version of exactly what our app-level mux approximates.
- Connection migration. QUIC connections are identified by a Connection ID,
not the 4-tuple, so a NAT rebinding or Wi-Fi→cellular handoff survives without
a new handshake. Our tunnel handles this today with reconnect + resume tokens
(
180.C.25.07NAT-change test); QUIC would make it seamless. - 0-RTT resume. QUIC/TLS 1.3 can resume a session and send application data
in the first flight (0-RTT). This overlaps with our resume tokens
(
180.C.25.03) but at the transport layer. - Firewall/NAT traversal on 443. Our hard requirement is reachability from a studio/home/coffee-shop network with no inbound ports. WSS on TCP/443 is essentially always allowed. QUIC is UDP/443, which a non-trivial fraction of corporate/hotel/guest networks block or throttle. Any QUIC adoption must keep a TCP fallback or we lose reachability — the entire point of the tunnel.
- Browser support for the Control Room. WebSocket is universally supported. WebTransport shipped in Chromium-based browsers and is available in recent Firefox; Safari/WebKit support is still limited/experimental. The Control Room is browser-first and must support Safari operators, so WebTransport cannot be the only control transport without excluding Safari.
- Operational maturity & cost. WSS terminates behind any standard L7 load balancer/reverse proxy (nginx, ALB, Cloudflare) with mature tooling. HTTP/3 termination is supported by these but is younger; UDP load balancing, MTU/PMTU discovery, and amplification-attack mitigation are additional operational surface for a small team.
- Don't fork the protocol. We have one protocol package and a working
app-level mux/scheduler. A second transport must reuse the same
RemoteCommandEnvelope, channel descriptors, and traffic classes, not fork them.
Considered Options#
Option 1: Adopt QUIC/WebTransport now as the primary control transport#
Replace the WSS tunnel with a WebTransport (browser) / raw-QUIC (host)
connection; map our seven channels onto native QUIC streams (one stream per
channel, or a stream per bulk object on file_transfer); use a QUIC datagram for
emergency_stop so it is never blocked by any stream.
Rejected for now. Three blockers:
- Reachability regression. UDP/443 is blocked on enough networks that making it primary would strand operators the WSS tunnel reaches today. We would have to ship both transports anyway, so QUIC-primary buys complexity without removing the WSS path.
- Safari Control Room. WebTransport is not dependable on Safari/WebKit yet. A browser-first Control Room that must run on Safari cannot drop the WebSocket control path.
- Operational immaturity for our scale. The team is shipping an MVP; adding UDP load balancing, HTTP/3 termination, and amplification mitigation now is cost we cannot justify before the control plane has real load.
Option 2: Reject QUIC/WebTransport permanently#
Commit to WSS-over-443 + the app-level mux forever; never revisit.
Rejected. The HOL-blocking limitation is real and structural: as we add
high-throughput file_transfer and richer events/stream_signaling traffic to
the shared TCP connection, a single lost segment in a bulk transfer measurably
delays control and event frames in a way no application code can remove. QUIC is
the correct long-term substrate for multiplexed control + media signaling, and
the migration cost is bounded precisely because we already modelled channels and
traffic classes as first-class. Closing the door now would be a mistake.
Option 3: Defer, behind an explicit adoption trigger, keeping the design QUIC-ready (chosen)#
Keep WSS-over-443 as the shipping transport. Treat QUIC/WebTransport as a planned second transport that we add when a measured trigger fires, not on a calendar date. Keep the protocol and the mux/scheduler transport-agnostic so the swap is a transport adapter, not a protocol rewrite.
This option is chosen.
Decision Outcome#
Defer adoption. Ship and operate the WSS-over-443 tunnel with the app-level
multiplexer (180.C.25.05) and priority/backpressure scheduler
(180.C.25.06). Do not build QUIC/WebTransport for the MVP.
Adopt QUIC/WebTransport as a second, negotiated control transport (never a silent replacement) only when any of these triggers is met and recorded with evidence:
- T1 — measured HOL pain. Production telemetry shows control/event-frame
delivery latency regressing under concurrent
file_transferload on the shared TCP connection, beyond the180.C.25performance budget, attributable to TCP-level HOL blocking (i.e. the scheduler already sent the frame but the socket delayed its bytes). - T2 — mobility pain. Reconnect/resume churn from NAT rebinding or network
handoff (
180.C.25.07NAT-change class) becomes a top operator complaint that QUIC connection migration would remove. - T3 — browser readiness. WebTransport reaches dependable support across the Control Room's required browser matrix including Safari/WebKit, removing the need to keep WebSocket as a parallel browser control path.
Adoption is additive and negotiated: the host advertises transport support in
its handshake/discovery TXT (180.C.25.01), the gateway picks the best mutually
supported transport, and WSS-over-443 remains the guaranteed fallback for
networks that block UDP/443 and for Safari. We will never have a configuration
that cannot fall back to the TCP tunnel.
Why defer rather than adopt or reject#
The benefit (removing TCP HOL blocking, seamless migration, 0-RTT) is real but not yet felt at MVP scale; the cost (dual transport, UDP reachability gaps, Safari gap, HTTP/3 ops) is immediate. Deferring captures the upside option while paying none of the cost now, and the "keep it QUIC-ready" rules below make the later swap cheap.
Keeping the Design QUIC-Ready (no-regret work to do now)#
These cost little today and make a later QUIC adapter a drop-in. The current code already satisfies most of them:
- Transport seam already abstracts the socket.
RemoteTunnelTransport(apps/bellona/remote-host/src/gateway-tunnel.ts) is a minimalconnect/send/close/onMessage/onCloseinterface. A WebTransport/QUIC adapter implements the same interface; the tunnel state machine, mux, and scheduler are unchanged. Keep this seam narrow — do not leak WebSocket-specific types into the tunnel/mux/scheduler. - Channels and traffic classes are protocol data, not transport data.
RemoteTransportChannel, the channel descriptors, and the eight traffic classes live inlibs/bellona/remote-protocol/src/channel.ts. A QUIC mapping reuses them verbatim: each channel → a QUIC stream;emergency_stop(our onlyunordered,neverBlockedchannel) → a QUIC datagram;file_transferbulk objects → one stream per object so loss is isolated per transfer. - Resume tokens are transport-agnostic.
RemoteTunnelResumeToken(libs/bellona/remote-protocol/src/transport.ts) is independent of how the bytes move; it maps cleanly onto QUIC connection migration + TLS 1.3 session resumption when we adopt. - Frame envelope already carries channel + sequence.
RemoteChannelFrametags channel/sequence/traffic-class explicitly, so over QUIC the per-stream ordering is native and oursequencebecomes a redundant integrity check rather than the ordering mechanism — no schema change required.
Migration Requirements (if/when a trigger fires)#
Recorded now so adoption is execution, not rediscovery. Each becomes its own
180.C.25.11.x task at adoption time:
- Transport negotiation. Extend the host handshake (
180.C.25.03) and mDNS TXT (180.C.25.01) with asupportedTransportslist (e.g.["wss-443","webtransport-h3-443"]). The gateway selects the best mutually supported transport and records the choice in session metadata + audit. WSS must always be in the list. - QUIC/WebTransport adapter. Implement
RemoteTunnelTransportover WebTransport (Control Room browser) and over a raw-QUIC library (host service, e.g. aquinn/msquic-backed Node/Rust binding). No changes to the tunnel state machine, mux, or scheduler. - Channel→stream mapping. Map each
RemoteTransportChannelto a dedicated bidirectional QUIC stream; mapemergency_stopto a QUIC datagram (or an urgent dedicated stream) so it is delivered without any stream's HOL blocking; map eachfile_transferbulk object to its own unidirectional stream so loss is isolated per object. Re-run the180.C.25.05HOL proof against the QUIC adapter. - Backpressure mapping. Replace the socket-buffer
bytesQueued()backpressure signal (RemoteSendChannel) with QUIC per-stream flow-control credit + connection-level flow control. TheselectNextToSendpriority logic and shed policy are unchanged; only thecanSend()source changes per stream. - 0-RTT vs replay safety. If 0-RTT is enabled, gate it to idempotent /
replay-safe traffic only.
emergency_stopandapprovalmust remain safe under 0-RTT replay; non-idempotentengine_command/file_transferinitiation must wait for 1-RTT confirmation. Reuse the existing idempotency-key dedupe window (180.C.25.03) as the anti-replay backstop. - Infrastructure / deployment. Add UDP/443 ingress + HTTP/3 termination to
the Hetzner (
180.C.25.08) and AWS (180.C.25.09) reference deployments: open UDP/443, configure HTTP/3 on the LB/proxy, set QUIC connection-ID-aware UDP load balancing, handle PMTU/MTU and the QUIC anti-amplification limit, and keep TCP/443 WSS listening in parallel. Add Alt-Svc advertisement so capable browsers upgrade. - Fallback + reachability probe. On connect, attempt QUIC and fall back to
WSS on UDP-block/timeout within a bounded budget, recording which transport
won per session for diagnostics (
180.C.25.12). Never strand a session that can only reach TCP/443. - Security review. Re-run the threat model (ADR-0002) for the QUIC path:
amplification, connection-ID linkability/migration privacy, 0-RTT replay,
datagram unreliability for
emergency_stop(must have an ack/retry at the app layer since datagrams are unreliable), and parity of the gateway-trust / E2E-encryption model (ADR-0003) across both transports. - Impairment test parity. Extend the
180.C.25.07impairment suite (apps/bellona/remote-host/src/network-impairment.ts) with a QUIC-adapter transport so every impairment (latency, loss, gateway restart, sleep/wake, NAT change, Wi-Fi disconnect, lid close) is asserted to recover over QUIC too — especially NAT change, which should now succeed via connection migration without a reconnect.
Consequences#
- The MVP ships on WSS-over-443 with the app-level mux + scheduler; no QUIC work is scheduled for the MVP.
- The transport seam, channel descriptors, traffic classes, frame envelope, and
resume tokens stay transport-agnostic so the later swap is an adapter, not a
rewrite. Code review must reject any WebSocket-specific type leaking past the
RemoteTunnelTransportseam. - When a trigger fires, adoption is additive and negotiated, WSS remains the
guaranteed fallback (UDP-blocked networks, Safari), and the migration-
requirements list above becomes the
180.C.25.11.xwork breakdown. - The
emergency_stopguarantee is explicitly carried forward: over QUIC it must use an app-layer ack/retry because QUIC datagrams are unreliable — this is called out in migration requirement 8 so the kill switch is never weakened by the move.