feat(server): place --target-device auto by GPU availability and multi-GPU technique

Merged
GR
5d ago
+696-42

Summary

--target-device auto now places a model by what the GPUs are doing at startup, and by how the model can use more than one GPU.

  • Busy GPUs are skipped. On AMD, a GPU is busy when another process has KFD queues there or more than 64 MiB in vram_<gpu_id> (/sys/class/kfd/kfd/proc). Free memory alone cannot tell: Strix Halo reports ~121 GiB free while holding DeepSeek V4 or half of a layer-split Qwen. A server single-GPU or split shows up on every GPU it spans, about a second after it starts and before its weights load. The startup log and --list-devices name it: busy=pid 1154989 luce_server Qwen3.8-27B-IQ4_XS-pure.gguf on hip:0+hip:1.
  • Free memory sizes each GPU (HIP reports Strix Halo's total as 1.25 TiB).
  • Placement order:
    1. a discrete GPU that holds the model alone;
    2. the architecture's multi-GPU technique over the free GPUs, when together they hold the model: DeepSeek V4/V4.1 expert parallel (--expert-device), layer-split families by layer (weights ∝ free memory). This targets models only Strix Halo can hold, where the discrete GPU's help beats Strix alone;
    3. one free GPU that holds it, discrete first;
    4. the roomiest free GPU for backends with expert offload;
    5. otherwise stop before loading, naming the busy processes.
  • With --profile, auto picks only the device, since a profile is qualified for one topology. --target-device auto --profile ds4-strix behaves as before.
  • Fix (separate commit): DeepSeek4's uncapped hybrid expert budget reserved 0.5 GiB on the main GPU, while check_device_headroom() requires 1.5 GiB on a discrete GPU. Every R9700 + Strix expert-parallel launch without LUCE_EXPERT_BUDGET_MB failed at 0.99 GiB free. The qualified profiles pin a lower budget, so they never hit it.

qwen4exp (Qwen3.8-Flash-Next) has no layer-split or expert-device path, so auto places it on one free GPU: the R9700 when it fits there, otherwise Strix Halo.

Validation (R9700 hip:0 + Strix Halo hip:1, rebased on c320db9a)

DeepSeek-V4-Flash-0731 ROCMFPX-MIX-STRIX (91.5 GiB), --target-device auto --max-ctx 8192, no profile:

R9700Strix HaloPlacementResult
freefreeexpert parallel hip:0,hip:1, peer access onloaded, 2.05 GiB free on R9700; answered "Paris"; routed calls 33% R9700 / 67% Strix
busy (Qwen 27B server)freeStrix Halo aloneanswered "Paris"; queues only on Strix, Qwen undisturbed
freebusynone: R9700 cannot hold it, DS4 cannot offloadrefused before loading
busybusynone: every GPU is busyrefused before loading

Why a model that fits the discrete GPU stays there: Qwen3.8-27B IQ4_XS (13.5 GiB) forced into an auto layer split ([0,13) R9700, [13,64) Strix Halo) decoded 15.1 tok/s over 128 tokens, against 36.2 tok/s on the R9700 alone. Both were warm, 2 runs each. With this order, auto keeps it on hip:0. The split path itself loaded and answered correctly.

Detection was checked against a Qwen 27B layer split (--target-devices hip:0,hip:1), a single-GPU Qwen server, and a server 2 s into loading.

  • test_server_unit 630/630
  • test_feature_gate, including new test_launch_policy cases for the availability policy
  • test/test_entrypoint.sh PASS

Not covered

  • Two servers launched within the same ~1 s can both see a GPU as free.
  • CUDA builds have no busy signal and fall back to free memory.
  • DS4 expert parallel picked by auto runs without the ds4-r9700-strix tuning. Its 33.7 tok/s on a 2-token reply is not a benchmark. ds4-strix on Strix alone is documented at 42 tok/s with 131K context. Whether DS4 should prefer Strix alone when both GPUs are free is open.

🤖 Generated with Claude Code