feat(pflash): unified view with the SSD session cache (stacked on #746)

Merged
GR
1d ago
+5331-24

Stacked on #746 (head eca7265bd). #746's branch lives on a fork, so this PR targets main and its diff also shows #746's commits. This PR's own commits are eca7265bd..HEAD, listed below. Merge it together with #746: it is the PFlash mechanism the PFlash v2 post describes and measures. The SSD session cache it builds on is already in main (#839).

What it does

Unified view, on by default with a trained head. PFLASH_UNIFIED=1 joins #746's release profile, so a server started with the head and probe GGUFs serves chats and agent sessions with one mechanism. PFLASH_UNIFIED=0 keeps #746's chat view.

  • What gets paged. Only the newest turn's long uncached content: the user messages since the last reply, or the outputs of the last tool calls (every tool by default, PFLASH_REGION_TOOLS). It needs at least --prefill-threshold (16K) droppable tokens and lines to cut at.

  • How it is cut. At line starts. A note says the content was shortened, and a marker says how many lines were left out. The request probe's pins (the question it finds and its recency pin) stay whole. History is never paged again.

  • Later turns. They continue the stored view token for token, so the prefix cache keeps working.

  • Recall. A new user question brings back left-out lines as excerpts at the start of its message, through a need-based gate: nothing comes back when the lines the view does not hold carry less than θ = 0.075 of the question's score mass (PFLASH_RECALL_GATE).

  • Hand-back. Requests the view cannot serve well go to the chat view and are served exactly as without the flag:

    • a cold uncached history;
    • an unreadable turn structure;
    • a remote drafter;
    • a scorer that gives no per-candidate scores (legacy, split, fold).

    A hand-back or a raw fallback ends the conversation's unified views, so no later turn revives a shortened history.

On the SSD session cache (--kv-cache-dir, #839):

  • Admission counts the disk. The restorable prefix is the deeper of the RAM snapshot and the deepest disk entry, in both views. It is an index lookup and reads no file.
  • Shortened conversations survive a restart and eviction. Views are kept in <kv-cache-dir>/.pflash-views, written when they leave RAM and at shutdown, and read on a RAM miss. A prompt served from a view gets a disk KV key: the tokens it served.
  • Limits. At most 256 views. The cache's TTL applies, counted from a view's last write and checked at every lookup.
  • Fail-closed. Every view file is checked in full before use, and one that does not load is deleted. Views of another model, template or context are never read.

With the disk cache off, decisions, records and log lines are unchanged.

R9700 checks

Setup:

  • One AMD Radeon AI PRO R9700, --kv-cache-dir with defaults.
  • Each restart check: shutdown (SIGTERM, flush), restart, next request.
  • Workloads are built from in-tree docs.
After a restartThis PRWithout the disk workFull prefill, cold
Raw chat, 27.3K tokens0.99 s, served raw, 27,277 positions from disk11.08 s, handed back and compressed whole29.07 s
Agent session, 45.1K tokens2.49 s, 44,721 positions from disk; answer and tool calls byte-identical to full prefill2.58 s56.22 s
Chat with a 27.3K paste paged to 7.8K, then a question4.69 s: view read from disk, 7,809 positions restored, recall brought back 864 tokens10.89 s, handed back and compressed whole29.10 s

Every chat answer was correct in every arm.

Release-profile default check. A launch with only the model files (head, probes, drafter) installs PFLASH_UNIFIED=1, the release threshold and curve, and gate 0.075. Its served prompt is byte-identical to a launch with the release environment set explicitly.

Final build (ac7ba4526), rebased onto #746. The paged chat and its post-restart question serve byte-identical prompts to the build before the rebase:

  • t1: 7,816 tokens served from 27,272;
  • t2: 8,794 tokens, with the view and 7,809 positions read from disk;
  • answer: correct.

TTFT was 10.4 s for t1 and 4.4 s for t2.

Known limit

The unified view does not keep #746's query pin (PFLASH_QUERY_PIN). The drafter computes that pin inside its own selection, and this view replaces that selection, so the pin is never applied here. The post's numbers were measured this way.

On the post's public R9700 runs (six sets plus LongCodeQA), the pinned text was cut in 19 of 228 paged requests. In none of those 19 did this view get wrong an answer that full prefill got right. Most of the cut pins were not the question: in LongCodeQA they land on the repository header, license or code of conduct. Passing the pin through to the view is a follow-up.

Tests

test_server_unit (HIP gfx1201 build, HIP_VISIBLE_DEVICES=-1): 994 passed, 0 failed, 6 skipped. They cover:

  • Unified view: paging, recall and the gate; hand-backs, including selections without candidate scores; probe pins kept; raw fallback ending the view; drafter failure with and without PFLASH_FAIL_CLOSED.
  • Release profile: it installs the unified view.
  • Disk: the restorable prefix rule, a restart with the history on disk in both views, the Full-scope-only rule, and a shortened chat continued across a restart.
  • Disk view store: restart, prefixes, forks, cap, lineages, identity, broken and inconsistent files, TTL at startup and at lookup, partial write.

Commits

  • a69a9cdea feat(pflash): format-agnostic region module for the unified view (stage 1)
  • 44469d925 feat(pflash): PFLASH_UNIFIED serves one view/recall mechanism for chat and agent turns
  • 0d36150e9 fix(pflash): unified view review fixes (FlowKV fence, re-read holds, reset view, single widening)
  • 82fa2b5a0 feat(pflash): need-based recall gate and fresh-minus-held recall source for the unified view
  • 6e5e469c7 feat(pflash): PFLASH_RECALL_TRACE_UNITS records each region's units for an offline recall replay
  • df0e9c69c fix(server): recall gate review fixes (fresh any-overlap, NaN fail-closed, new-region drops, served deficit)
  • e87ac2000 fix(server): unified view takes every user message since the last reply as the newest turn
  • 678a85284 feat(server): unified view release defaults, hand-back to the chat view
  • 55c923fc3 feat(pflash): admission counts the disk session cache, in the unified view too
  • 6755e53b8 feat(pflash): shortened conversations survive a restart and eviction on the SSD cache
  • 9e88f91c1 feat(pflash): the release profile serves the unified view
  • 3ffd0fa6e fix(pflash): unified view review fixes before it ships as the release default
  • ac7ba4526 fix(pflash): unified view keeps views until a hand-back is served, and checks repeats and disk views

🤖 Generated with Claude Code