Share a pinned FFmpeg dependency bundle across gallery CI shards

Merged
FA
3d ago
+1707-10

Standard gallery shards independently install FFmpeg and 86 dependencies from Ubuntu mirrors. In PR #494, five shards finished setup in 15–27 seconds while three remained in package installation for over 17 minutes before starting any gallery tests.

Add one blocking dependency producer, parallel with the shared-wheel build. It downloads the reviewed Ubuntu package bundle once on a cache miss, verifies committed SHA-512 digests and sizes, and publishes one same-run artifact. All eight standard shards restore that artifact and install offline, with integrity checks before privileged installation and an H.264 encode preflight. The producer owns an exact lockfile-keyed cache; warm runs verify cached packages without contacting package servers.

Package identities come from authenticated Ubuntu indexes. The committed maintenance command refreshes the lock without changing host APT state. Dependency downloads, producer setup and consumer restoration have bounded timeouts. Workflow/selection checks require the producer, fail the aggregate result on its failure, and reject reintroduced per-shard APT downloads. Specs, a release note and reproducible local setup measurements are included.

Validation:

  • 203 focused dependency, workflow and selection tests passed, including corruption/symlink/path rejection, cache reuse, offline consumer wiring, summary failure propagation and mutation checks.
  • Full-worktree pre-commit hooks, Ruff lint/format, focused typechecking, Python-floor validation, actionlint and workflow verification passed.
  • Verified the actual 87-package bundle and deterministic lock refresh against signed Ubuntu metadata. Local cold download: 55.05s; warm verification: 0.10s.
  • Installed the bundle and encoded H.264 in an isolated Ubuntu 24.04 fixture with networking disabled: 4.37s. Raw observations and reproduction commands are committed under spec/.

Hosted run 40485681835 passed the new cold producer: one upstream bundle download and verification took 9.65s, followed by one artifact upload. All eight shard restoration/install/encoding steps passed in 7–17s. The artifact identity and individual step timings are committed in spec/benchmarks/ci-gallery-dependencies.json. The full CI suite continues independently; these observed setup timings are not a throughput guarantee.

Hosted warm-cache run 40486201516 then restored the same exact cache key: package verification took 0.15s and made no Ubuntu package-server requests. Producer log: https://github.com/reflex-dev/xy/actions/runs/40486201516/job/122129019366

Summary by CodeRabbit

  • Bug Fixes
    • Gallery CI jobs now share a pinned, integrity-verified FFmpeg package bundle, reducing repeated package downloads and enabling offline restoration across all eight gallery shards.
    • Gallery dependency failures are included in the overall CI result, helping ensure issues are reported consistently.
  • Documentation
    • Added guidance for dependency verification, lockfile updates, and reproducing the gallery setup without network access.