fix(spf): serve relay pulls whose fill cannot be reset reliably
Summary
A v10 publisher could not be watched from any relay other than the one it publishes to. When moq-relay pulls a track that already has content, its SUBSCRIBE asks for a fill. The publish session refused that SUBSCRIBE with NOT_SUPPORTED, because the browser's WebTransport writers have no commit(). A relay pulls a track a second time whenever a viewer joins from another relay, so the catalog never reached that viewer. This change serves the subscription and still fails the fill, as the draft requires.
The problem, measured on our fleet
The fleet is four moq-relay 0.17.0 relays (oci1.{phx,ord,iad,fra}.relay.mux.global) in a star with iad as the hub. The test is a v10 MoqPublishMedia publisher at phx and a v10 <simple-moq-video> player, both in headless Chromium 147 on one host, with glass-to-glass measured canvas to canvas.
-
At
mmcc/moq-publisher(d65c0c6ba, anddafbe32d6before it), the player plays when it is on the publisher's relay (phx/phx 3/3, TTFF 0.26–0.31 s). On any other relay it stays atreadyState 0with no catalog: phx→iad, phx→ord and phx→fra failed 9 of 9 today. Last nightdafbe32d6failed the same paths 84 of 84. -
The player is not the cause. The same player build plays phx→fra from two other MSF+LOC publishers (relay-sdks and media-shim, 3 of 3 each).
-
The publisher's relay logs the refusal for every failed run. Below is phx, for one phx→iad run:
lite::publisher: track info started track=catalog # iad asks phx about the track ietf::subscriber: subscribe started track=catalog # phx pulls it from the v10 publisher ietf::subscriber: subscribe cancelled track=catalog # +14 ms: the probe is done lite::publisher: subscribed started track=catalog # the viewer's real subscribe ietf::subscriber: subscribe started track=catalog lite::publisher: subscribed error track=catalog err=unsupported # +19 msphx logged
err=unsupportedin every one of today's failed runs (94 lines across 8 of them). iad reportsinternal errordownstream andtrack info failedon every retry.
Root cause
- moq-relay (0.14.17 through 0.17.0) subscribes upstream with Next Object, plus
FILL_PARAMETERSfor the current group (subscribe_joininmoq-net/src/ietf/subscriber.rs). - A cross-relay viewer makes the publisher's relay pull each track twice:
- The first pull is a TRACK_INFO probe. It arrives before the track has content, so its fill range is empty and it is served. Binding it replays the catalog, so the track now has a Largest Object.
- The relay then cancels the probe and pulls again for the viewer. That fill range is now nonempty.
#handleIncomingSubscribeanswered a nonempty fill withREQUEST_ERROR NOT_SUPPORTED "fill requires reliable stream reset"whenever the control writer had nocommit(). Chromium 147 hasWebTransportbut noWebTransportWriterand nocommit(), so every second pull was refused.- A viewer on the publisher's own relay causes only one pull, before there is content, which is why same-relay playback worked.
- The local two-relay mesh played for the same reason. On loopback the real subscribe arrives 2 ms after the probe, before the probe's catalog write has completed. On the fleet it arrives one relay round trip later (67–82 ms at phx for an iad viewer).
- By the same code path (inferred, not measured), a same-relay viewer who joins after every earlier viewer has left is also refused, since moq-relay 0.15+ drops an upstream subscription once nothing uses it.
- The relay hub that the cross-engine tests use always gave stream writers a
commit(), so no test exercised the browser's case.
Changes
- SUBSCRIBE: the session serves it whatever the fill. A nonempty fill is still failed as §5.1.3.1 asks: open the fill fetch stream, write the FETCH_HEADER,
commit()where the writer has it, then reset. - Without
commit(): the reset may discard the header, and the subscriber then sees no fill at all. That leaves its join at the next group boundary, just as a failed fill does. For this publisher that costs nothing: groups never span bindings, and every bind replays the current group from object 0 under a fresh Group ID. moq-relay 0.17.0 waits on a fill only to stitch a mid-group tail, which this publisher never sends. - REQUEST_UPDATE: an update that requests a fill is acknowledged and failed the same way, instead of ending the subscription.
- Tests:
createRelayHub({ reliableReset: false })models a transport withoutcommit().
Stacked on #62
This branch is now based on #62 (fix/moq-publish-done-track-ended at 10606e84e), which ends tracks with PUBLISH_DONE and an exact Stream Count. Only the top commit belongs to this PR, and it merges after #62.
The two PRs met in #openAndResetFill. #62 counted a fill fetch stream (subscriber.fillStreams++) just past the commit() gate that this PR removes. The fill now counts wherever the session opens one, whether or not its header survives the reset:
- The draft counts streams opened. Draft-20 §10.12 defines Stream Count as "the number of data streams the publisher opened for this subscription, ... including any fill fetch streams". The publisher knows that number exactly, so the 2^64−1 "unknown" value does not apply.
- An overcount only costs a timeout. The draft expects a subscriber to come up short sometimes: subscribers "SHOULD use a timeout or other mechanism ... in case the publisher ... reset a stream before the SUBGROUP_HEADER". An undercount is the dangerous error: "If a subscriber receives more streams for a subscription than specified in Stream Count, it MAY close the session with a PROTOCOL_VIOLATION." Leaving the fill out would undercount every time its header does arrive.
- #62 counts media streams the same way. Its track publisher counts a data stream once it commits to writing the stream's header, so a stream that is reset later still counts.
- moq-relay 0.17.0 closes the session in neither case. It ends the track downstream once it has read as many stream headers as Stream Count says and no fill is outstanding, or after a 1 s grace (
poll_settledinrs/moq-net/src/ietf/subscriber.rs,GRACEinrs/moq-net/src/tail.rs). A fill whose header was lost stays outstanding, so that track end waits out the grace whatever the count. That 1 s delay before "track ended" is the cost of a reset withoutcommit(). It comes from reading the relay's code; on the fleet it did not appear in 21 unpublish runs (the fill headers reached the relay before the reset), so the header-lost case is still unmeasured.
New test: counts a fill reset without reliable reset in the PUBLISH_DONE Stream Count, with the fill's header delivered and with it lost. Both cases report 3 track streams plus the fill, so 4. Both fail if the count is gated on commit() (they get 3), and both fail on #62 alone, where the SUBSCRIBE is refused.
Testing
- New session tests in
publish-session.test.ts:- serves a relay's second pull of a live track without reliable reset support: the relay's exact probe → cancel → subscribe sequence and parameters. It checks SUBSCRIBE_OK, that the binding moves to the new alias, that the fill stream is reset, and that a REQUEST_UPDATE fill is acknowledged.
- still resets the fill when the reset may discard its header: the peer sees only the reset, and the subscription stays up.
- Updated test:
relay-hub.test.tsruns its relay-pull case with and without reliable reset. - Red, then green: both new session tests and the new hub case fail on the base and pass with the change.
- Suites, on the rebased head:
pnpm -F @videojs/spf testpassed 3 runs in a row (169 files, 2,727 tests, 11 skipped).pnpm build:packages,pnpm lintandpnpm typecheckpass. - Fleet, rebased head (
f1067b323, 18:39–18:54Z, fleet on its default config): the player at target 0.5 s passed 9/9 with the same G2G as before the rebase (phx/phx 554 ms, phx→iad 608 ms, phx→fra 688 ms; TTFF 0.31 / 0.89 / 2.10 s). A relay-sdks Python subscriber decoded the broadcast (video and audio) on phx/phx, phx→fra and two rejoin cases, 12/12, with every frame decoded. - Track ends on the fleet, with #62 underneath: after
unpublish(), viewers got PUBLISH_DONE status 2 "track ended" on catalog and video in 9/9 player runs, about 46 / 211 / 514 ms after the stop at phx / iad / fra, and the relays logged no errors. One thing still to look at: with audio in the broadcast, 5 of 12 subscriber runs saw one track (audio in all 5, video once) end with status 0 "internal error" although the publisher sent status 2. It looks like a race with the WebTransport session closing 24–40 ms afterunpublish()(not yet confirmed). Closing the page withoutunpublish()still ends every track with status 0. - Fleet, before the rebase: the table below was measured on
7e1a9002e(this change without #62). - Fleet: the before and after builds were bundled the same way and run with the same harness: 3 repeats per cell,
target-latency0.5 s (the default) and 0.1 s, on 2026-10-08 between 14:52 and 15:20Z, with the fleet on its default config. Each cell shows passes, the median G2G p50 (range) and the median TTFF (range). The last row is a control on two local moq-relay 0.17.0 relays clustered over loopback.
| Path | Before (d65c0c6ba), target 0.5 s | After, target 0.5 s | After, target 0.1 s |
|---|---|---|---|
| phx/phx (same relay) | 3/3 · 559 ms (549–561) · TTFF 0.26 s (0.26–0.31) | 3/3 · 563 ms (562–566) · TTFF 0.34 s (0.31–2.98) | 3/3 · 164 ms (156–166) · TTFF 0.25 s |
| phx→iad | 0/3, no catalog | 3/3 · 615 ms (612–623) · TTFF 0.94 s (0.91–1.21) | 3/3 · 211 ms (191–221) · TTFF 0.86 s (0.85–1.14) |
| phx→ord | 0/3, no catalog | 3/3 · 620 ms (615–647) · TTFF 1.09 s (1.08–1.49) | 3/3 · 230 ms (214–262) · TTFF 1.04 s (1.02–1.49) |
| phx→fra | 0/3, no catalog | 3/3 · 727 ms (703–839) · TTFF 2.10 s (2.03–2.58) | 3/3 · 311 ms (298–377) · TTFF 2.10 s (1.93–2.47) |
| local two-relay mesh (loopback) | 3/3 · 559 ms (545–571) · TTFF 0.15 s (0.14–0.15) | 3/3 · 551 ms (541–578) · TTFF 0.15 s (0.14–0.18) | – |
- Relay logs: with the change, the publisher's relay logged 0
err=unsupportedlines for the cross-relay runs. - Join time: with the change, the cross-relay join is 0.9–2.1 s. For comparison, last night the older
e3c77e8ebuild (moqt-19) joined in 2.1–2.5 s at similar G2G (phx→iad 612 ms, phx→fra 719 ms at 0.5 s). - The 2.98 s outlier: one phx/phx run took 2.98 s to its first frame. This change does not touch that path, and the relay logged both of its subscribes 35 ms apart, as in the other runs. Where the time went is not yet known.
🤖 Generated with Claude Code