feat(files): make expires_at the truth for file content (DW-992)

TO
5d ago
+711-78

Bottom layer of the DW-992 stack (single-clock content retention). Stands on its own; the retirement-rule layer follows.

Why

The API shows an expires_at on every input file, but nothing enforced it:

  • content stayed downloadable past the date, and the row was only marked expired as a side effect of being read;
  • output and error files had no expires_at at all, and kept reporting processed after their content had been dropped with a retired archive week;
  • expires_after[seconds] on upload was uncapped (and silently ignored when it didn't parse), so a file could record a date content retention would never reach;
  • a batch created from an expired input was refused by the repository and surfaced as a 500.

Content retention is moving to one creation-anchored clock for prompts and completions. That only works if the API cannot show a date the content will not reach.

What

One setting governs all three files of a batch. batches.files.default_expiry_seconds (previously an unused key defaulting to 24 h; now 30 days, which is what uploads already received) is the lifetime given to an upload without expires_after, and to a batch's output and error files measured from the batch's creation. max_expiry_seconds caps what an upload may request. Both reach the repository through a new FileExpiry setting on PostgresRequestManager.

  • Upload: expires_after[seconds] outside [min, max] (1 h to 30 d by default) or unparseable → 400, and nothing is stored (counted in dwctl_upload_rejections_total as invalid_expiry). Sent before the file part, it is refused before the file is read; sent after, the file is read first and the upload is then aborted, as with purpose. Same range as OpenAI.
  • Output and error files are created with expires_at. Ones created before this read as created_at + default, derived on read, with no backfill of existing rows.
  • GET /files/{id}/content past expires_at (or status expired) → 410, matching how deleted results files already answer.
  • GET /batches/{id}/results → 410 once the output or error file is past expires_at, alongside the existing 410 for a deleted results file. It reads both files through the repository, so older results files get the same derived date.
  • POST /batches: an input that is not processed → 400. The repository's own ValidationError on a deleted, expired or unprocessed source now maps to 400 instead of 500. There is no separate check that the input outlives the completion window; OpenAI does not make one either.
  • Expiry config (min/default/max positive and ordered) is validated whether or not batches.enabled is set. The request manager is built from these keys on every instance, so a misordered config is now a config error instead of a panic at startup.
  • config.yaml and docs/src/reference/configuration.md document the three keys: the 30-day default, the upload range, the 400 and 410 responses, and when results files expire.

Output and error files expire default_expiry_seconds after the batch is created. OpenAI counts from when the batch completes, so ours can expire up to one completion window earlier. The docs say so.

Not changed: files.status is still only written on read (check_and_mark_expired); nothing deletes content here. That is the next layer.

Tests

Ten new tests, all against Postgres:

  • Repository: results files carry the batch's clock; legacy results files derive the default; an upload beyond max is refused and leaves no row.
  • Uploads: the default lifetime, using a non-default default_expiry_seconds so the test shows the setting reaches fusillade; range rejection, including the bounds.
  • Downloads and results: 410 on download past expires_at. 410 from /batches/{id}/results when either results file has a past date, and when the derived date has passed. Both paginated and streaming paths are covered.
  • Batch creation: results files show created_at + default under a non-default config. An expired input gets 400. That test runs the repository's ValidationError → 400 mapping.
  • Config: a misordered or non-positive expiry is rejected with batches and the daemon disabled.

Full just test rust: 4,296 passed.