feat(dwctl): count upload refusals by code and measure decode-slot waits

Merged
CR
6h ago
+368-20

Closes DW-994. Follow-up to DW-985 (#2082, #2083).

Compressed request bodies and compressed batch uploads shipped with good coverage for realtime refusals: unsupported_content_encoding, body_decode_failed and request_too_large go into onwards_rejections_total, and the gateway rejection alerts pick them up. Two failure modes from that work had no signal. This PR adds one for each.

Gap 1: file-upload refusals weren't counted by reason

Batch uploads don't go through the inference body layer. upload_file refuses them through dwctl's own Error: a corrupt .jsonl.gz gets a 400, an xz/bzip2/zip/lz4 file gets a 415, and so does a body that isn't multipart/form-data. None of these reach onwards_rejections_total, and dwctl_http_requests_total only gives the status, so they were mixed in with every other 400 and 415 on the files route.

New counter: dwctl_file_upload_rejections_total{code}. It covers every FileUploadError variant, plus the refusals upload_file makes before it reads the file:

codestatuscause
not_multipart415body isn't multipart/form-data
upload_capacity429the upload queue is full, or no upload slot came free in time
batch_disabled403the batch API is switched off for the account
invalid_multipart400multipart content type, but the form can't be read (e.g. no boundary)
stream_interrupted400multipart stream broke mid-upload
file_too_large413over limits.files.max_file_size, from either Content-Length or the stream
too_many_lines400over max_requests_per_file
invalid_json / invalid_utf8400a bad line
invalid_compression400starts as gzip/zstd but is corrupt or truncated
unsupported_compression415xz, bzip2, zip, lz4
no_file / empty_file / invalid_purpose400form contents
model_access_denied403a line names a model the caller can't use
validation_error / unprocessable_line400 / 422per-line validation
image_unfetchable422an image origin refused access
image_transient / image_store_failed / image_access_unavailable / image_fetch_failed503image normalisation failed on our side or upstream

Every code is registered at 0 when the Prometheus recorder is installed, so the first refusal of a kind is a 0 → 1 step that increase() can see.

Why a dedicated counter rather than rejections::refuse. I looked at recording these through the existing rejection machinery so they'd land in onwards_rejections_total. It doesn't fit well:

  • The files route is outside the onwards router, so the Recorder extension isn't installed there. refuse also works on a Response, but the handler returns dwctl's Error.
  • The labels don't apply. model would always be empty, because one upload spans many models. traffic only distinguishes realtime from dispatched inference, so an upload would be labelled realtime inference traffic.
  • onwards_rejections_total counts client errors. Several upload failures are 503s caused by our image store or an image origin. Mixing them in would make a server problem look like a client one.
  • GatewayRejectionNew and GatewayRejectionSpike are tuned for inference traffic. A separate series lets upload alerts get their own thresholds.

The cost is that the existing dashboard and alerts don't pick the new series up on their own, so they need adding there.

Gap 2: decode-slot waits weren't measured

Decoding a compressed realtime body takes one of DECODE_SLOTS, one per CPU. Requests over that bound wait rather than fail, so if the slots saturate (a burst of compressed traffic, or abuse, since decoding runs before authentication) compressed requests just get slower, and nothing measured that.

in_slot now records:

  • dwctl_inference_decode_slot_wait_seconds (histogram): time from asking for a slot to getting one. Buckets are 1 ms to 5 s with an edge at 0.25 s, so a 250 ms threshold reads an exact bucket. A free slot is taken at once, so almost every observation lands in the first bucket.
  • dwctl_inference_decodes_waiting (gauge): decodes waiting for a slot right now. It's set to 0 at startup. A drop guard decrements it, so a request dropped while waiting leaves the gauge and records no wait.

Behaviour is unchanged: requests still wait and are never refused, and the permit still moves into the blocking task.

Tests

  • test_every_upload_refusal_is_counted_by_its_code: builds every FileUploadError variant, refuses it under a local recorder, and checks it's counted once under its own code, and that the code is in the zero-registered list.
  • test_upload_rejection_codes_start_at_zero: every code renders at 0 after initialisation.
  • test_upload_capacity_refusals_are_counted: holds the only upload slot and checks the refused upload's 429 is counted as upload_capacity.
  • The existing end-to-end tests for a truncated gzip, an xz file, non-multipart bodies, a broken multipart form and an org with batch disabled now also check that each refusal moves its counter.
  • records_how_long_a_decode_waited_for_its_slot: uses a paused tokio clock, so it's deterministic with no sleeps. It holds the only slot, checks the gauge reads 1, advances the clock 300 ms, releases the slot, and checks the wait lands between the 0.25 and 0.5 buckets. A decode that found a free slot lands in the first bucket.
  • a_request_dropped_while_waiting_leaves_the_queue: dropping a waiting decode takes the gauge back to 0 and records no duration.

just lint rust and just test rust pass.