feat: overlay targets are scheduling hints, not a serving class

Merged
HA
17h ago
+341-354

What

A serving class is where a request is routed. Until now, an organisation overlay's targets made the request resolve as a custom class. This PR removes custom: overlay targets become scheduling hints, exactly like account-file priority and targets, and the request keeps its routed class (standard today).

BeforeAfter
Overlay targets โ†’ resolved class custom (analytics resolved_serving_class, tariff and cache lookup)Resolved class is always the routed class (standard)
Targets sent through a separate path, even to members that refuse scheduling fieldsTargets, priority and account priority all go through one hints path, only to dynamo members that accept scheduling fields
Overlay class_pricing could price customclass_pricing prices one of the model's classes (standard). custom is refused by validation, and rows the catalog owns under it are retired on the next apply

Why

  • Classes are destinations. model is standard and, later, model:fast is fast. Customer deals (priority, limits, price, tolerations) sit on top of whichever class a request hits. Latency targets are customer labels for alerting and diagnostics; nothing routes on them. custom was left over from the earlier target-routing design.
  • Fixes a pricing gap. With targets on an overlay, the price lookup used class custom and skipped an account's standard-class price, falling back to the model default. The new e2e test overlay_targets_change_neither_the_class_nor_the_price covers it.

Changes

  • onwards:
    • ServingClass has only Standard. ServingTargets, the overlay targets field and the separate stamping path are removed. A stored config that still carries overlay targets loads, and the field is ignored.
    • Hints are not sent to a named-pool member that refuses scheduling fields, matching how its nvext is already stripped.
    • The request span records onwards.serving.priority and onwards.serving.targets (whether latency targets were sent), separately from onwards.serving.resolved.
  • dwctl:
    • The sync maps model_overlays.targets (ttft_ms, itl_ms, priority) onto the overlay's hints.
    • PricingClass is standard only; custom joins the retired class names.
    • The batch-in-class check that standard made redundant is removed.
  • dashboard:
    • Pricing lists the model's classes only. A class price is shown winning over the organisation's all-class price, as billing does.
    • Overlay details label targets as scheduling hints.
  • docs: the overlay how-to, the configuration reference and the class-identity note.

Compatibility

  • Existing overlay files keep working: targets: {ttft_ms, itl_ms, priority} is unchanged on the wire and now applies as hints.
  • An overlay file that uses class_pricing: custom fails validation (startup and the offline validator). Move such a price to standard.
  • No migration.
  • onwards drops public items (ServingTargets, ServingClass::Custom). Its next release is already a major.

Tests

  • just lint rust, just lint ts: green.
  • just test ts: 805 passed.
  • just test rust: 4244 of 4245 passed. The failure was test_e2e_ai_proxy_streaming_chat_completions_with_fusillade_header, a known load-timing flake; it passed 3/3 on its own and in the previous full run.
  • After merging main (deps/CA bundle): onwards 676 passed, and the affected dwctl suites passed.

๐Ÿค– Generated with Claude Code