feat(dwctl): count upload refusals by code and measure decode-slot waits
Closes DW-994. Follow-up to DW-985 (#2082, #2083).
Compressed request bodies and compressed batch uploads shipped with good coverage for realtime refusals: unsupported_content_encoding, body_decode_failed and request_too_large go into onwards_rejections_total, and the gateway rejection alerts pick them up. Two failure modes from that work had no signal. This PR adds one for each.
Gap 1: file-upload refusals weren't counted by reason
Batch uploads don't go through the inference body layer. upload_file refuses them through dwctl's own Error: a corrupt .jsonl.gz gets a 400, an xz/bzip2/zip/lz4 file gets a 415, and so does a body that isn't multipart/form-data. None of these reach onwards_rejections_total, and dwctl_http_requests_total only gives the status, so they were mixed in with every other 400 and 415 on the files route.
New counter: dwctl_file_upload_rejections_total{code}. It covers every FileUploadError variant, plus the refusals upload_file makes before it reads the file:
| code | status | cause |
|---|---|---|
not_multipart | 415 | body isn't multipart/form-data |
upload_capacity | 429 | the upload queue is full, or no upload slot came free in time |
batch_disabled | 403 | the batch API is switched off for the account |
invalid_multipart | 400 | multipart content type, but the form can't be read (e.g. no boundary) |
stream_interrupted | 400 | multipart stream broke mid-upload |
file_too_large | 413 | over limits.files.max_file_size, from either Content-Length or the stream |
too_many_lines | 400 | over max_requests_per_file |
invalid_json / invalid_utf8 | 400 | a bad line |
invalid_compression | 400 | starts as gzip/zstd but is corrupt or truncated |
unsupported_compression | 415 | xz, bzip2, zip, lz4 |
no_file / empty_file / invalid_purpose | 400 | form contents |
model_access_denied | 403 | a line names a model the caller can't use |
validation_error / unprocessable_line | 400 / 422 | per-line validation |
image_unfetchable | 422 | an image origin refused access |
image_transient / image_store_failed / image_access_unavailable / image_fetch_failed | 503 | image normalisation failed on our side or upstream |
Every code is registered at 0 when the Prometheus recorder is installed, so the first refusal of a kind is a 0 → 1 step that increase() can see.
Why a dedicated counter rather than rejections::refuse. I looked at recording these through the existing rejection machinery so they'd land in onwards_rejections_total. It doesn't fit well:
- The files route is outside the onwards router, so the
Recorderextension isn't installed there.refusealso works on aResponse, but the handler returns dwctl'sError. - The labels don't apply.
modelwould always be empty, because one upload spans many models.trafficonly distinguishes realtime from dispatched inference, so an upload would be labelledrealtimeinference traffic. onwards_rejections_totalcounts client errors. Several upload failures are 503s caused by our image store or an image origin. Mixing them in would make a server problem look like a client one.GatewayRejectionNewandGatewayRejectionSpikeare tuned for inference traffic. A separate series lets upload alerts get their own thresholds.
The cost is that the existing dashboard and alerts don't pick the new series up on their own, so they need adding there.
Gap 2: decode-slot waits weren't measured
Decoding a compressed realtime body takes one of DECODE_SLOTS, one per CPU. Requests over that bound wait rather than fail, so if the slots saturate (a burst of compressed traffic, or abuse, since decoding runs before authentication) compressed requests just get slower, and nothing measured that.
in_slot now records:
dwctl_inference_decode_slot_wait_seconds(histogram): time from asking for a slot to getting one. Buckets are 1 ms to 5 s with an edge at 0.25 s, so a 250 ms threshold reads an exact bucket. A free slot is taken at once, so almost every observation lands in the first bucket.dwctl_inference_decodes_waiting(gauge): decodes waiting for a slot right now. It's set to 0 at startup. A drop guard decrements it, so a request dropped while waiting leaves the gauge and records no wait.
Behaviour is unchanged: requests still wait and are never refused, and the permit still moves into the blocking task.
Tests
test_every_upload_refusal_is_counted_by_its_code: builds everyFileUploadErrorvariant, refuses it under a local recorder, and checks it's counted once under its own code, and that the code is in the zero-registered list.test_upload_rejection_codes_start_at_zero: every code renders at 0 after initialisation.test_upload_capacity_refusals_are_counted: holds the only upload slot and checks the refused upload's 429 is counted asupload_capacity.- The existing end-to-end tests for a truncated gzip, an xz file, non-multipart bodies, a broken multipart form and an org with batch disabled now also check that each refusal moves its counter.
records_how_long_a_decode_waited_for_its_slot: uses a paused tokio clock, so it's deterministic with no sleeps. It holds the only slot, checks the gauge reads 1, advances the clock 300 ms, releases the slot, and checks the wait lands between the 0.25 and 0.5 buckets. A decode that found a free slot lands in the first bucket.a_request_dropped_while_waiting_leaves_the_queue: dropping a waiting decode takes the gauge back to 0 and records no duration.
just lint rust and just test rust pass.