PR #839 - Luce-Org/lucebox

What this does

A long coding-agent session can lose its RAM snapshots in three ways:

another session evicts them;
the server restarts;
the user switches model.

luce_server now keeps those sessions on disk, and the disk cache is on by default. A session whose RAM snapshot is gone restores its raw KV from disk and prefills only the new suffix. This works both while the server runs and after a restart.

Defaults and flags

Flag	Default	Meaning
`--kv-cache-dir <path>\|off`	`$XDG_CACHE_HOME/lucebox/kv-cache`, else `~/.cache/lucebox/kv-cache`	Cache directory. `off` (or `--no-kv-cache`) disables the cache.
`--kv-cache-budget <MB>`	`131072` (128 GB)	Hard cap on all snapshot files, with LRU deletion. Clamped to the free space minus a reserve (the larger of 1 GiB and 5% of the disk, at most 16 GiB).
`--kv-cache-ttl <hours>`	`72`	Deletes files unused for this long. `0` = never.
`--kv-cache-shutdown-flush <s>`	`120`	Longest time a graceful shutdown spends writing RAM snapshots. `-1` = no limit.

The other disk flags are unchanged: --kv-cache-min-tokens, --kv-cache-interval, --kv-cache-cold-max and --disk-prefix-cache (default full). /props → full_cache reports disk_enabled, disk_budget_mb and disk_ttl_hours.

Dashboard: Luce-Org/luce_box#391 (branch feat/session-cache-ssd-setting) still proposes 16 GB, off by default. Its default should change to match this PR: on, 128 GB. That repo is not touched here.

When something is written: only when it leaves RAM

Nothing is written after a turn. A snapshot is written in two cases:

RAM eviction. A new snapshot takes the slot of an older one (its reservation's victim), the resident budget evicts several to make room for a large one (below), or the budget evicts one after a commit. This applies to the single-sequence path and to the concurrent (paged) scheduler.
Graceful shutdown (SIGTERM/SIGINT). Shutdown is one last eviction of everything still resident, most recently used first. It is bounded by --kv-cache-shutdown-flush. One log line gives the sessions, edit backups not written, files, MB and seconds, plus anything the limit dropped.

Caveat: a crash, kill -9 or power loss loses whatever was only in RAM. States written before that still restore.

An active session that stays resident writes nothing. That matters because a 90-120K-token Qwen3.8-27B session with q4_0 KV is 1.5-2.2 GB per snapshot. The 30 s idle write-behind of earlier revisions is gone.

What is persisted

Each conversation's newest snapshot: the inline snapshot of its last request, or the agent-turn state with the generated reply (tool path).
The shared system/tools heads.

Edit-backup rule. Since #835, a new turn keeps the previous turn's snapshot in RAM so that an edit of the last message can resume there.

That kept turn is not written while a deeper snapshot of its conversation is resident (PrefixCache::persist_on_evict). It only serves an edit, which the deeper file cannot, and writing it would double the bytes per conversation.
Once the deeper snapshot is gone, the kept turn becomes the conversation's newest snapshot and is written normally.
Pruned turns are never written.
Writing a state deletes the same session's shallower files on disk, except the head.

Compressed prompts (PFlash, FlowKV) are not raw history and never get a disk key.

How it works

Writes. Eviction hands the snapshot buffer to a writer thread without a copy, and the request continues at once.

The snapshot is indexed immediately and served from memory until its file exists.
Queued payload is bounded at 4 GiB; past that, the evicting request waits, and the wait is logged.
Each write goes to <file>.tmp, is fdatasynced and renamed, and the directory is synced.
A failed write (for example, a full disk) is logged and skipped.

Restore. With the full policy, a lookup probes three keys: the whole prompt, every chat boundary, and the deepest stored key that is a prefix of the prompt. These are index lookups; only a hit reads a file.

A disk state deeper than the best RAM hit wins over it. A session evicted to disk usually still matches the shared head in RAM. If the read fails, the RAM hit is used.
The paged scheduler reads the file on a reader thread while live streams keep decoding.
Each file stores the tool-call replay text (ToolMemory), and startup loads it back. Without it, a new process would render earlier tool calls differently and no deep state would match.

Safety: fail closed.

Format v5 ends with an XXH64 checksum of the whole file. A corrupt or partial file is a logged miss, is deleted, and the prompt is prefilled.
Startup deletes temporaries, files of older formats and truncated files.
A file restores only when its identity matches:
- the model's path, size and mtime, plus a content fingerprint (SHA-1 of the first and last MiB);
- the KV types and K rotation;
- --max-ctx, the chat template and the draft model;
- every tensor's name, type and shape. Qwen also checks each tensor against the live cache and the paged page table.
Each file stores the exact key token ids, and a hit compares them with the prompt.
One server per directory, enforced by an exclusive lock.
LUCE_DISK_CACHE_DIGEST=1 logs an XXH64 of every restored state, tagged ssd or ram.

A large capture evicts several older entries (own commit). Upstream skipped a capture when no single evictable entry freed enough room. A long session that arrived after many short ones (subagent threads) was then never captured, so it reached neither RAM nor disk. Now a capture that fits the budget at all plans several evictions, least recently used first: edit backups first, then leaves. The rule is the same on the single-sequence and paged paths.

Upstream used one victim so a capture never disturbs the committed cache before it lands. The plan keeps that: it is all or nothing and decided before anything is evicted. Its entries leave RAM (to disk) only when the capture commits, so a capture that fails or is cancelled evicts nothing.
The plan never takes protected pins, heads that other entries extend, the capture's own ancestors or restore source, or entries that another capture in flight writes, evicts or counts as freed.
A capture larger than the whole budget is still skipped (budget_skips).

Pre-existing bug fixed (qwen35). snapshot_adopt() never set the snapshot layout, so every Qwen disk restore was rejected. The caller then decoded on an empty cache, which produced garbage output. Adopted snapshots are now dense, and a failed restore fails the request.

Commits

Folded into exactly 3 commits on main (5f402c61f), including the review follow-ups. The fold preserves the verified final tree.

#	Commit	Change
1	`5ee22b574`	feat(server): persist evicted sessions to disk with a budget, TTL and…
2	`5f744cd83`	docs(server): document the disk session cache, its defaults and evict…
3	`d6774ae46`	fix(server): harden disk cache lifecycle, scope and capture budgets

Review fixes and validation (13 Oct)

Corrupt tensor metadata is rejected before GGML tensor construction: unsupported types, invalid quantized dimensions, overflowing shapes and out-of-file payloads become logged misses. The original corruption regression reproduced a floating-point exception; four malformed metadata cases now pass during live lookup and restart.

The follow-up fixes also disable a failed cache initialization so RAM serving continues without an unstarted writer queue; refresh duplicate-save TTL timestamps on disk; respect request disk scope for agent-turn saves and concurrent reads/writes; reap completed abandoned reader threads during serving; and provide an exclusive Windows directory lock and native Windows default cache directory. /props.full_cache documents the effective disk enablement, budget and TTL. The disk write-failure regression is portable to Windows and root-owned test runs.

The reported snapshot_adopt / restore_and_generate_impl failure is fixed here: adopted Qwen snapshots are dense, and restore failure returns an error before decoding from cleared KV. A CPU-backed test uses the real Qwen backend to load a .dkv, adopt it, restore exact tensor values, and reject an incompatible restore without generating tokens.

The shared-ancestor reservation concern did not reproduce: two capture orders are tested at tight budgets. Reservations retain each capture's full pending bytes, so the shared ancestor is not credited twice. No accounting workaround was added.

Validation on the final tree:

HIP gfx1201 builds: luce_server, test_server_unit, test_recurrent_snapshot.
SSD branch unit suite, GPUs hidden: 687 passed, 0 failed, 2 skipped.
Recurrent snapshots: 5 passed, 0 failed.
Combined with the updated #746: 917 passed, 0 failed, 6 skipped, plus all 5 recurrent snapshot checks.
Direct regression: shortened snapshots remain reusable in RAM and receive no disk key during eviction or shutdown; raw sessions still persist.

CUDA and Windows builds and R9700/Strix GPU CI passed on the preceding published head; the DGX job was cancelled before running steps. CI is rerunning for these follow-ups. This review did not run a new live GPU workload; the hardware measurements below are the earlier gates.

Numerics

SSD restore vs RAM restore of the same state: byte-identical.

0 digest differences over 38 paired SSD restores in the final gate run (c4, with the multi-entry eviction fix), across the single-sequence path (including tool-call agent-turn states) and the paged path.
c2 and c3 (27 pairs) give the same.
Greedy answers are identical for every request restored at the same depth.

A RAM restore and a fresh prefill can still differ, because prefill runs over different chunk boundaries. That is a prefix-cache property, unchanged here. In the gate, every answer that differs from the reference belongs to a request restored at a different depth.

Gate results: Modal L40S

Setup:

Plan frozen before serving, with readings pre-registered as labels, not vetoes.
Qwen3.8-27B IQ4_XS, q4_0 KV, max-ctx 131072, AR decode, temperature 0.
Public MIT ggml corpus.
The cache dir was on the container's local disk (gVisor overlay).
All timings are L40S timings. The page-cache drop before each restart is not verified to take effect under gVisor, so the read times are container-disk times, not cold-NVMe claims.

Reading	Result
R1: writes on eviction	WRITES-ON-EVICTION. Single-sequence: 10 eviction writes; paged: 12; 0 write failures.
R2: same depth as RAM	c3: DEPTH-DIFFERS(9), every one a state the RAM budget never captured (one-victim rule), so it was never on disk. c4, with the multi-entry eviction fix: none differ. All 9 requests (for example the 32K-token thread's 36698/39172) now restore at the reference depth.
R3: numerics	BYTE-IDENTICAL. 0 digest differences (c3: 27 pairs; c4: 38).
R4: TTFT, cold prefill / RAM follow-up / SSD follow-up after graceful restart / reference RAM restore of the same follow-up	15.8K tokens: 7.61 / 0.76 / 0.98 / 0.30 s (read 628 MB in 0.21 s) 44K: 22.97 / 1.04 / 1.10 / 0.91 s (1124 MB, 0.32 s) 88K: 64.30 / 0.77 / 2.03 / 1.60 s (1895 MB, 0.61 s)
R5: disk bytes per evicted state	~18.4 KB per token, plus a fixed ~340 MB single-sequence or ~140 MB paged. 554 MB at 11.6K tokens, 833 MB at 27.5K, 1.9 GB at 88K. Each state is written once.
R6: TTFT during a write burst	c2: NO-STALL. Six 40-70K prompts that each evict the previous one; disk-on/off TTFT ratio median 1.001, max 1.009. c4 (fix): median 1.003, but one request is over the 1.15 line, so the pre-registered label is STALL-SUSPECT. That request is B3-probe at 0.99 vs 0.86 s. The 70K B3 state, skipped before the fix, is now captured: it pushes out 1.6 GB, and the next probe pushes it out again (1.6 GB written while that probe prefills).
R7: graceful restart	RESTART-RESTORES. Shutdown flushes took 0.44 s for 1.5 GB and 1.60 s for 5.4 GB, well under the 120 s limit; the process stopped in 1.1-2.2 s. Restarts restored from SSD for 6/6, 3/3 and 5/6 requests in c3; in c4 for 6/6 and 6/6.

The gate found one bug. On the single-sequence path, a RAM hit on the shared 512-token head ended the restore search. A session evicted to disk was therefore never restored while the server ran: 0 SSD restores in the first run. It is fixed in this PR, with a unit test that fails without the fix. After the fix, the same group made 9 SSD restores, all byte-identical where paired.

Unit tests: test_server_unit 679 passed, 0 failed, 2 skipped on this branch (HIP build for gfx1201, GPU hidden). The suite covers:

TTL and the LRU budget across restarts;
checksum and corrupt-file rejection;
identity mismatch;
persist-on-evict and the edit-backup rule;
shutdown flush, then restart;
a deeper disk state winning over a RAM head hit;
agent-turn eviction writes;
a large capture evicting several entries, written to disk, on the single-sequence and paged paths (disk on and off); one larger than the budget skipped; protected, head, ancestor and in-use entries never taken; edit backups first and not written; a failed or cancelled capture evicting nothing.

R9700 restart check

One AMD Radeon AI PRO R9700, 12 Oct, built with the PFlash release branch merged in (release defaults on). A 45K-token
agent session (public MIT ggml tool calls and results): turn 1, graceful shutdown, restart, turn 2.

	Turn 2 time to first token	Restored from disk	Answer
Disk cache on, after restart	2.58 s	44,721 of 45,136 tokens (1.1 GB read in 0.29 s)	byte-identical to full prefill
Full prefill, cold	56.2 s	none	reference

Paths not covered

States the RAM budget cannot hold. A capture larger than the whole budget, or one whose only possible victims are protected or in use, is still skipped. It is then neither in RAM nor on disk. In c4 this happened once: the last agent-turn save of a restart arm, next to its own restore source.
Crash or kill -9. Loses what was only in RAM, by design. Not exercised.
Single-sequence --paged-attention and the concurrent qwen4exp engine. They keep their checkpoints outside the backend's snapshot slots and run without a disk cache (logged at startup).
Unit-tested only, not served: TTL and disk-budget LRU eviction, the free-space clamp, corruption, identity mismatch, the second-server lock, and the edit-backup exclusion (the gate workload had no edits).
Not exercised: R9700/HIP serving beyond the one restart check above, the Auto/Fixed disk scopes, non-Qwen backends, and compressed prompts.

PFlash integration limit. The combined #839 + #746 unit suite passes, but compressed PFlash views are not persisted to SSD. This PR excludes shortened snapshots from disk keys; it restores raw session states and ToolMemory replay metadata. Disk-aware PFlash admission and compressed-view restart persistence require a separate integration change.

🤖 Generated with Claude Code