improvement(ci): collect idle node_modules sticky disks at 3 days
What's going on
The Docker layer-cache caps we shipped last week worked — layer caches gave back 431 GB. But total storage barely moved, because node_modules sticky disks grew +350 GB over the same period and ate the win. They're now 44 disks / 601 GB (~$305/mo) and the single largest storage line, with total storage back over the 500 GB quota.
Why they grew — checked, not assumed
Two possible causes, and they need different fixes:
- Disks accumulating cruft over their life → the fix would be in
test-build.yml, not retention. - More disks alive at once → the fix is retention.
It's the second. Disks created yesterday are already 14.8 GB, and the whole population now clusters tightly at 14.7–14.9 GB rather than the bimodal spread it had before. They're born that size — no accumulation. What changed is churn: bun.lock moved 24 times in 14 days on staging alone, and disks are created at ~6.3/day.
Steady state is new disks/day × GB/disk × retention:
| predicted | observed | |
|---|---|---|
| 7d (today) | 606 GB | 601 GB |
| 3d (this PR) | 260 GB | — |
The model reproduces reality to within 1%, so the projection is trustworthy.
Why 3 days
Not a judgement call — the data has a cliff:
| Retention | Disks kept | GB kept | Reclaimed now |
|---|---|---|---|
| 7d | 41 | 558 | 43 GB |
| 4d | 34 | 458 | 144 GB |
| 3d | 20 | 277 | 348 GB |
| 2d | 19 | 263 | 362 GB |
Going 4d→3d reclaims another 204 GB; going 3d→2d reclaims only 14 GB more. So 3 days captures essentially all of the win, and it clears a long weekend — a PR whose last CI run was Friday afternoon still hits a warm disk on Monday.
Expected: ~$173/mo steady-state, ~$176/mo reclaimed on the first run.
Why age-based and never PR-triggered
The key contains no PR identifier. Every open PR whose checkout has the same bun.lock mounts the same disk — there are far more open PRs than distinct lockfile hashes (28 hashes across 44 disks), so sharing is the common case. A pull_request: [closed] hook would destroy a disk many other open PRs are actively using, most aggressively for the most-shared disk.
Safety — every guard was executed, not reasoned about
| Test | Result |
|---|---|
| Normal run at 3d against the live listing | 24/44 selected, 348 GB |
retention_days: 0 | refused |
retention_days: "3d" (non-numeric) | refused |
| Every disk crafted to look idle | refused — that's a listing/clock fault, not idleness |
A turbo-cache-build-push key injected into the listing | 0 selected |
Two independent filters: a server-side --search, then a local regex re-proving the full key shape. The event segment is [a-z_]+ rather than an enumerated push|pull_request — the key interpolates ${{ github.event_name }}, and a workflow_dispatch disk already exists that an enumerated list would have skipped forever.
Other choices worth a look in review:
- GitHub-hosted runner, so collection keeps working during a Blacksmith outage or
CI_PROVIDERbreak-glass switch — exactly when disks idle and still bill. - CLI pinned by version + SHA256, verified against the vendor's published
.sha256sidecar, notcurl | sh; the job holds an org-wide token. Auto-update disabled so the pin holds. permissions: {}, no checkout.- Group by key before the staleness test, since
deletewithout--archremoves every arch variant.
Worst case of a wrong delete is one cold bun install — and the bun cache disk is not lockfile-keyed, so it never idles, is never collected, and a refill links from a warm store with no network fetch.
Setup required before this does anything
- A GitHub org admin runs
blacksmith org-token create --label ci-stickydisk-gc --organization simstudioai(printed once). - Add it as repository secret
BLACKSMITH_CLI_TOKEN. The job fails with a clear message if it's missing. - Must land on
main—scheduleonly runs the default-branch copy. - First run via Actions → Sticky Disk GC → Run workflow → dry_run = true; confirm every listed key is a
node-modules-<hash>one.
Type of Change
- Improvement
Testing
actionlint clean including its shellcheck pass. The selection logic was executed against the live account listing and against crafted listings for each guard (table above). Pinned CLI URL returns 200 and its SHA256 matches the published sidecar. Diff is one YAML file with no JS, so the repo audits run in CI.
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing
- No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)