feat(api-keys): stop storing API key plaintext (DW-1012)

Merged
TO
10d ago
+1430-1718

Part 6 of 7 for DW-1005 (DW-1012). Keys authenticate by secret_hash and fusillade dispatches by key id, so the stored plaintext copies are now only a leak risk. Nothing reads or writes them any more; DW-1013 (stacked on this) drops them.

Merge gates

Merge only when both hold in production:

  • dwctl_api_key_pending_reveals_needing_plaintext reads 0 on every pod.
  • No fusillade request without a key id is still live (rows queued before #2156 have finished).

What changes

  • API keys: the secret of a new or rotated key is returned once and only its hash is kept. api_keys.secret stays NOT NULL for the previous release; new rows take a column default ('unset-' || gen_random_uuid(), unique and not a key), so no code names the column.
  • Fusillade: realtime, flex, background and batch work stores no key; request_templates_g2.api_key defaults to ''. Dispatch always uses the key id; a row without one carries no credential and the loopback refuses it.
  • Reveal: works only from the wrapped pending secret. A pending key without a usable wrap returns 500 and stays revealable; rotating it fixes it.
  • Reconcile: no longer wraps from plaintext; still re-wraps old wraps and clears orphaned ones. dwctl_api_key_pending_reveals_unrecoverable replaces ..._needing_plaintext; dwctl_api_key_reveals_total drops its source label. Update any dashboards or alerts on those.
  • Migrations (dwctl and fusillade 20261105105806): the two column defaults. Metadata-only.
  • Existing plaintext stays until DW-1013.

Rollout

Previous-release pods read both columns as non-null. They only ever see the defaults for new rows, which they store but never use (dispatch by key id, reveal from the wrap). Rolling back to the previous release is safe.

Testing

cargo nextest run -p dwctl -p fusillade -p fusillade-core -p fusillade-arsenal --all-features (3757 passed) and just lint rust pass locally.