feat(runtime): worker validation gates

Closed
SE
19d ago
+586-6

Part 2 of 6 of the worker validation gates stack (DEP ai-dynamo/dynamo#15897). Based on part 1. Together with parts 3–6 it replaces #2–#5, cut down from ~4,950 to ~2,000 lines.

Summary

A worker can run checks against itself before it reports ready, so a broken worker stalls a rollout instead of serving.

DYN_WORKER_VALIDATION_GATES='{"smoke": {"exec": ["python3", "/checks/smoke.py"], "timeout_secs": 900}}'
  • Each gate runs once, in parallel, when the worker's first local engine registers (part 5 moves this to the loopback endpoint). Exit 0 is a pass; otherwise the end of stderr is the reason. A gate past its deadline fails and its process is killed.
  • /health reports 503 with each gate's state and reason until every gate passes. /live is unchanged, so a held worker isn't restarted.
  • Verdicts live in memory: a restarted container is checked again, and a failed gate holds the worker until the pod is deleted.
  • Startup fails on bad config, without the system status server, or in sidecar mode.

Unset, nothing changes.

Validation

Config parsing and error table, a failing gate holding the worker, the gate deadline, /health held while /live stays 200 for passing and failing gates, and the status-server requirement.

🤖 Generated with Claude Code