#843 feat(server): PFlash prefill compression under concurrent serving (stacked on #746)

Stacked on #746. GitHub shows #746's commits too: review only this PR's two commits, feat(server): PFlash prefill compression under concurrent serving and fix(server): make concurrent PFlash thread-safe and bound its memory (a merge commit brings in #746's latest head). Retarget onto main once #746 merges.

What it does

With #746, --paged-attention refuses PFlash, so a server with --max-concurrency > 1 runs every long prompt as a full prefill. This PR lets dense Qwen3.5 serving use PFlash while it serves several requests at once:

• A compression thread beside the slots. The scheduler hands a prompt over the compression threshold to a thread of its own. The drafter scores it there while the live slots keep decoding. The job then returns to the head of the queue, behind any other prepared jobs, with the compressed prompt. The prefix cache stays on the scheduler thread, which measures the prompt's restorable prefix before the hand-off; the compression thread never reads it.
• Memory for the drafter. The drafter scores beside the resident target, so skip-park is forced. The paged pool's automatic budget leaves free the drafter's weights plus 56 KiB per prompt token up to --max-ctx (kv_pool_reserve_bytes). 56 KiB rounds up the measured 53 KiB/token peak (33K–206K-token prompts, R9700).
  An explicit --kv-pool-tokens must leave that reserve free too, or the server refuses to start and says how many tokens fit.
• Admission. Raw prompts are admitted up to --max-ctx, the rule #746 uses on the serial path, and the reserve covers that length. Longer prompts get a 400. Routed requests (multi-model serving) are compressed like the others. A routed prompt the drafter has already scored waits for that model instead of reporting busy, so the router never has it scored twice.
• Usage and timings. usage.prompt_tokens counts the client's prompt, streaming or not. usage.timings.pflash carries the compression stats, as on the serial path.
• Thread safety. The compression thread and the scheduler thread share no mutable state:
  • The drafter resolves its cache's KV types without setting and unsetting LUCE_KV_TQ3. setenv could reallocate the environment while the other thread reads it.
  • The last-error string is per thread, so neither thread can free or replace the other's error.
  • On HIP, graphs computed without the qwen4exp profile leave the process-wide MMB/HC bookkeeping alone when it is empty. qwen4exp behavior is unchanged.
• Guards. The server checks four things at startup:
  • the release profile's strict selection with the trained head's scorer (PFLASH_SCORING_HEAD_GGUF; no legacy or split scorer), the configuration the memory reserve was measured with;
  • the drafter released after each prompt: --draft-residency persistent is refused, because its retained scoring sessions would sit outside the reserve;
  • an in-process drafter, since the IPC drafter is single-request only;
  • a dense Qwen3.5 target: the feature gate allows paged attention with PFlash only for qwen35 with --max-concurrency > 1, so DS4 stays refused and qwen4exp concurrency keeps its previous behavior.

The serial path is unchanged. Serial PFlash on qa8 measured 78.5 s on the first commit and 78.0 s on the head, run back to back (77.3 s on the #746 head in an earlier session).

Results

Setup:

• Hardware and models: R9700 (32 GB), Qwen3.8-27B IQ4_XS target, Qwen3.5-0.8B BF16 drafter.
• PFlash: release profile (trained head plus request and segment probes).
• Server flags: --max-ctx 139264, q4_0 KV.
• Concurrent arms: --paged-attention --max-concurrency 4, with 4 requests in flight (8 for c8).
• Every item was served in every arm, with 0 errors.

workload	serial PFlash	concurrent full prefill	concurrent PFlash
qa8: 8 × ~22K-token QA, short answers	77.3 s wall, TTFT 31.4 s mean, 8/8	379.2 s, 185.8 s, 8/8	94.6 s, 37.8 s, 8/8
cd4: 4 × 84K-token code-debug, long reasoning	346.2 s, 151.3 s, 4/4	1304.1 s, 1154.8 s, 4/4	340.0 s, 163.0 s, 4/4 *
mix8: 4 long QA + 4 short chats	49.7 s; short TTFT 13.5 s mean (max 28.8)	199.2 s; short 5.4 s	53.5 s; short 0.5 s (max 0.6)
qa8 with 8 in flight	77.5 s, 43.7 s	–	86.5 s, 49.2 s, 8/8 †

Concurrent PFlash numbers are from the head of this branch. Serial and full-prefill numbers are from the #746 head.

† From a back-to-back run against the first commit, which measured 86.5 s and 49.1 s. One earlier single run of the head measured 93.7 s, so single runs of this arm vary by about 7%.

* In one earlier run of the same design, one cd4 reasoning reply hit its 2048-token cap and missed. Its served prompt (21,742 of 83,939 tokens) was identical to the serial run's. That is variation in batched decoding, not a compression difference.

Edge case, one 136,684-token prompt at the context limit:

• It compresses to 31,113 tokens.
• Device memory peaks at 29.28 of 31.86 GiB.
• usage.prompt_tokens reports 136,684.
• A 190K-token prompt is refused with 400.

What these numbers mean:

• Against concurrent full prefill, which is what --max-concurrency 4 serves today, concurrent PFlash is 3.7–4.0× faster in wall time, with the same correctness.
• Short requests start in 0.5 s beside long prompts. On the serial path they wait 13.5 s on average behind the long ones.
• Against serial PFlash on one GPU, long prompts start later. Long-prompt mean TTFT is 8–20% worse on the long-only workloads, and 33% worse on mix8, where the short chats run beside the long prompts. Wall time ranges from 2% better (cd4, where batched decode pays) to 22% worse (qa8). Two causes:
  1. A paged prefill is slower than the serial one: a lone 22K prefill takes 31–33 s against 21 s.
  2. The drafter shares the GPU with the slots, and any target step slows it (about 4× beside a prefill, 6–9× beside decode).

Scheduling the drafter differently does not change the total GPU work. I tried several variants: making long prefills or every target step wait while the drafter scores, and having the drafter and long prefills take turns in arrival order. None beat this version on mean TTFT across the workloads, so none are included. One trade-off may be worth an opt-in flag later: making long prefills wait while the drafter scores cut cd4 wall time to 260 s (against 340 s here), but raised qa8 mean TTFT to 47.1 s (against 37.8 s).

Limits

• Dense Qwen3.5 only. DS4 and the IPC (remote) drafter are refused under concurrent serving.
• Drafter residency. The drafter loads and frees for each prompt, as auto residency does on the serial path; --draft-residency persistent is refused. Keeping it resident would need its retained scoring sessions in the reserve.
• Pool size. The reserve shrinks the paged pool, which all slots share. On the R9700 with 4 slots, the pool holds 456K tokens at --max-ctx 139264 (557K without the reserve) and 252K at --max-ctx 204800.
• Disk cache. Compressed prompts are never written to the disk prefix cache. The full-cache restore that serial PFlash does before scoring does not run on this path.
• Visibility. Prompts waiting for the compression thread do not show on /status.

Possible follow-ups

• A higher HIP stream priority for the drafter, so decode steps slow it less.
• Faster paged prefill: this is the main gap to serial PFlash.

Tests

• test_feature_gate covers paged attention with PFlash: allowed for concurrent qwen35 on CUDA and HIP, refused at --max-concurrency 1 and for DS4.
• test_admission checks that the reserve formula covers the five measured drafter peaks.
• test_server_unit adds four scheduler cases on a mock engine:
  • a long prompt on an idle server is served compressed, with usage.prompt_tokens equal to the client's prompt;
  • a decoding chat keeps stepping while the drafter scores;
  • two long prompts arriving together are both compressed;
  • a routed long prompt is compressed and served by the model that scored it.
• test_server_unit also checks that the last error is per thread. test_kv_quant checks KV-type resolution without the TQ3 shorthand.
• Each new scheduler and error test fails with its fix reverted.
• Full test_server_unit: 944 passed, 0 failed, 4 skipped. test_feature_gate: 4 passed.
• On the R9700, the server refuses to start with --kv-pool-tokens 557056, saying that at most 456144 tokens fit. It also refuses --draft-residency persistent.

🤖 Generated with Claude Code