PR #30496 - manaflow-ai/cmux

Summary

Restore fuzz (chaos diskfull) found two ways a full disk under cmux-tui's state leaves the person with nothing to do and no reason:

h2 (STATUS 3149). The local cmux-tui died while its disk was full. Its reconnect's server ensure failed. After the disk had space again, the app started no new daemon by itself (none within 60 s), and in 2 of 3 runs it did not offer Restart cmux-tui. A window that showed content had no Restart control at all. Pressing Restart (debug.daemon {action: restart}) recovered in about 1 s.
h3 (STATUS 3213). On a full disk the daemon refused the person's New Workspace (operation.failed, "database or disk is full"). The app correctly kept no row, but showed no refusal, toast or dialog.

After this change:

The app keeps starting its local daemon. The local reconnect and first connect (RetryPolicy.localReconnect, .localFirstConnect: keepsTrying) keep one attempt per 30 s backoff cap past their 10 timed retries, on the injected clock, instead of waiting for an event only. A disk that has space again sends no event. App activation, Retry Now and Restart still wake the loop at once and start the spacing over. Remote machines and every other loop keep event-only waits.
Every failed start offers Restart cmux-tui, with the reason. EnsureOutcome records each failed ensure as a DaemonStartFailure (disk_full when the error says so or the state volume has under 64 MB free, else failed). DaemonRecovery.offersRestart is true for any failed start, not only a stuck owner. The automatic restart still needs stuck-owner failures (UnresponsiveOwnerWatch is unchanged). debug.daemon reports start_failure and start_failure_detail.
A window that shows content gets the way out. DaemonRecoveryNotice shows one until-closed toast (the standard CmuxToastCenter, so the toast's own motion gate and keys apply) in the workspace window the key or main window acts for. It says why cmux-tui is not running ("cmux-tui could not start: the disk is full. Free some space and cmux starts it again by itself.", or a short generic reason), and its button is Restart cmux-tui, which asks the same CmuxDialogSpec confirm as the launch window's button. It ends when the daemon is back, a restart starts or the offer ends. A cancelled confirm shows it again. Closed by the person, it stays closed until the daemon connects again.
A refused person's New Workspace shows one refusal. It reads "New Workspace did not apply: the disk is full" through the existing action.effect.failed path. A scripted run (CLI, socket, MCP) still gets the typed daemon error and shows no HUD (ActionRegistry.trackOrReport(_:as:)).

No daemon state is deleted or reset.

Cause (on the base, 88dcb8942b6)

h2, no automatic restart: RetryPacer.swift:92 returns nil after RetryPolicy.reconnect's 10 timed retries (RetryPacer.swift:32). The reconnect loop then waits for a socket change or an explicit event only (DaemonConnection.swift:400).
h2, no Restart offer: DaemonRecovery.swift:116 offers Restart only when the last ensure looked stuck or the daemon does not answer. DaemonLauncher+Replace.swift:84 counts only restore_timeout, timedOut and server.unavailable as stuck, so a spawn that failed on a full disk offered nothing.
h2, no control over content: WindowController.swift:240 shows the connecting view, which holds the only Retry and Restart controls, only when the window has no content.
h3: AppActions+Workspaces.swift:163 handed the create's failure (:175) to registry.track, which keeps it only for a capturing caller (ActionRegistry+Work.swift:75-76). A keyboard, menu or palette run captures nothing, so the failure was dropped. The red below proves it: the action ran (perform true), the stand-in daemon received create-workspace and refused it, and the refusal HUD count stayed 0.

Changed assertion in an existing test

DaemonRestartTests.restartIsOfferedOnlyAfterAPlainEnsureFailedAndReplacesThenConnectsAgain asserted at :176 that Restart is hidden after a start failure that is not a stuck owner (server.spawn_failed). That assertion encoded this bug: a full disk fails exactly that way and left nothing to press. It now expects Restart shown. Its wait after Retry Now waits for Retry Now's own ensure, because Restart already shows before the press. The rest of the test (the replace runs the same ensure plus --replace-unresponsive, then connects again with a plain ensure, one replace only) is unchanged and passes.

Testing

Reds (each fails on its own assertion on feat with the tests only):

h2: CmuxNextAppTests.DaemonRecoveryAfterFailedStartsTests at 4d96f95d6c93, job ebaff5a8a29162f4c7189400: offersRestart false after a disk-full ensure (:103), debug.daemon offers_restart false (:115), no reason exposed (:117); the reconnect parked for an event after 10 failed starts, did not reconnect, and stopped at 11 starts (:208, :212, :213).
h3: CmuxNextAppTests.NewWorkspaceDiskFullRefusalTests at 71665d876f89 (branch fuzz-red/restore-h3-red), job b0cc51eb433543f2a3f59099: the person's create reached the daemon and was refused, and no refusal showed (:91, :93, :95, :96). The scripted test passes on the base, as it should: it guards that a caller keeps the typed error.

Greens: all 8 suites below passed (8 passed, 0 failed), 5 of 5, at da8798baaa71 (head): 4eb957f0ce1924f5d3579bef, 1e64070fbc06abd921ce7893, c97b5b57e4f1019e01d73392, ba699c66be77e730685205db, cd94fc92fe25605088cf1ed5. Suites: DaemonRecoveryAfterFailedStartsTests, DaemonRecoveryNoticeTests (new: a window with content shows the notice with the disk-full reason and Restart; Restart asks first; a cancelled confirm shows it again), NewWorkspaceDiskFullRefusalTests, DaemonRestartTests, DaemonAutoRestartTests, BusyRefusalReconnectTests, RetryPacerSocketBudgetTests, UnresponsiveOwnerWatchTests. The same 8 suites also passed 5 of 5 at 4c8a6c2da3ed (1ec96cb8, 28b51b2e, bde4ea06, 7d87de79, 5fdd58f2), before the last commit replaced English-text assertions with the localized string and the typed start_failure.

DaemonRestartTests alone on the base 88dcb8942b6 passed (job 7f107cbf400e92fe232f0f64): the one red it had at fff3cc7a6ef9 came from this branch's offer change and is fixed.

Compile: fleet compile-check CmuxNext with tests at da8798baaa71 passed (940c168b2ab8f1e126d18a9c); CmuxNext + app at 4c8a6c2da3ed passed (10b6bc8ad7719763d8f0b86d); Release app compile at 4c8a6c2da3ed passed (1c5b9112ecd1b3d74f2464ec). prepush-gate passes on every commit.

Localization audit: three keys, each with all 21 locales its catalog carries, every one translated (no copied English): refusal.daemon.diskFull ("the disk is full", CmuxNextDaemonService/Resources/Refusals.xcstrings), the one disk-full reason shared by h2 and h3; daemon.recovery.diskFull (a format string around it) and daemon.recovery.stopped (CmuxNextDaemonRecovery/Resources/Localizable.xcstrings). scripts/localization_catalog.py check reports no error for these keys. ./scripts/localize-changes cannot prepare CmuxNextDaemonRecovery (the package has several string catalogs), so I wrote those entries by hand. The tool's other reports (debt over the baseline in CmuxNextActions, CmuxNextSettings and other catalogs) are base debt in files this PR does not touch.

Observed / Not run

Observed: the reds and greens above, run on the fleet; the compile-checks; prepush-gate.

Display proof at the head (the real app on a display worker, through the mini scheduler, exclusive host):

Display-worker suite job 6bc9adc8699e6e31d2c11401 at da8798baaa713367b4e9a18079ddcb8fbb49677d (gui-session-v1, cmuxs-Mac-mini-3, console session, cmux-ci run --display, CMUX_TEST_REQUIRE_GUI=1), exit 0. All 8 PR suites passed, 31 tests, none skipped: DaemonRecoveryAfterFailedStartsTests 2, DaemonRecoveryNoticeTests 1, NewWorkspaceDiskFullRefusalTests 2, DaemonRestartTests 3, DaemonAutoRestartTests 2, BusyRefusalReconnectTests 6, RetryPacerSocketBudgetTests 3, UnresponsiveOwnerWatchTests 12.
Build job 74bf492462d9ec4492560f1c (head da8798baaa71). Bundle h2h3fix-da8798ba, commit da8798baaa713367b4e9a18079ddcb8fbb49677d.
h2: round fgh-h2-head-4202, capture-slot lease slot-cmux19-2ee9e0291d4a on cmux19-perf, restore fuzz chaos diskfull with daemon kills, --visible, harness 855dff1a8d3 (#30531). 23 judged, 13 daemon kills, 0 violations, app_daemon_differ 0. After a daemon kill on the full disk, the window showed "disconnected" with its content (66 samples, no blank pane). When the space came back, the daemon came back by itself in 18.2 s with no Restart press. On the base it never did.
h3: same build, round fgh-h3-head-4203, lease slot-cmux22-0c6402a26e98 on cmux22-perf. The person's New Workspace (debug.toast perform newTab, origin user) on the full disk showed exactly one refusal, "New Workspace did not apply: the disk is full". No ghost workspace stayed, now or after a relaunch. The 4202 probe gave the same result.
STATUS 3393.

Not run:

A real capture of the new notice.
DaemonCrashLoopTests (needs the real cmux-tui binary; keepsTrying keeps its crash-loop socket limit, but I did not run it).
The tests of every other ActionRegistry user.

Read only: the h2 fuzz runs that failed with server.unavailable (1 of 3) also offered Restart before this change; the cause of that one ensure failure is not known.

Changelog

Fixed: When cmux-tui stops and cannot start again (for example because the disk is full), cmux says why, offers Restart cmux-tui in the open window, and starts it again by itself once it can; a New Workspace refused on a full disk now says "did not apply: the disk is full"

Proof

No capture: the new notice is a standard CmuxToastCenter toast (no new view or style). It shows only after the local cmux-tui died on a full disk, and the capture minis cannot reach that state yet. DaemonRecoveryNoticeTests checks its text, action and confirm. The lead's diskfull fuzz round on a fleet build of this head checks it in the real app.

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Fixes two restore-fuzz findings where a full disk under cmux-tui's state left the person with no visible state and no way out: the app now keeps starting its local daemon, offers Restart cmux-tui with the reason in any open window, and shows a refusal when a person's New Workspace is refused.

Daemon recovery (h2)

The local reconnect and first connect keep one attempt per 30 s backoff cap past their timed retries, so the daemon restarts by itself once the disk has space again.
Every failed start offers Restart cmux-tui; a window that shows content gets an until-closed toast naming the reason (a full disk is told apart) with the same Restart confirm as the launch window.
debug.daemon reports start_failure and start_failure_detail.

New Workspace refusal (h3)

A person's New Workspace refused on a full disk shows one "did not apply: the disk is full" refusal; a scripted run still gets the typed daemon error and no HUD.