improvement(file-parsers): parse spreadsheets on a worker thread
Merged
WA
19d ago+681+681+012345678901234567890123456789-7-7-0123456789
Summary
XLSX.readis synchronous, so every spreadsheet parse throughparseBufferblocked the event loop for the whole parse (1.1–2s on 16–23 MB workbooks on Bun)- Spreadsheet buffers (
xlsx/xls/xlsm/xlsb/xltx/ods) now parse on aworker_threadspool running the sameXlsxParser; output is identical - Pool is bounded to one worker (in-process parses were already serialized by the sync parse, so peak workbook memory stays where it is); jobs queue, idle workers terminate after 30s
- Input bytes are copied into a fresh
ArrayBufferand transferred, so the caller'sBufferis never detached; options cross without the signal - Abort: a queued job is dropped, a running job's worker is terminated and the call rejects with
signal.reason(same value the in-process path throws) - Errors rebuild as the same class (
FileParserErrorwith code/message/cause name+message,ZipBombError,ArchiveIntegrityError), sogetFileParserErrorCodecallers see no change - Safe fallback: missing bundle, load failure, startup that never reports ready, or a crash mid-parse disables the pool for the process (one warn) and runs the job in-process exactly as before
- Enabled only for the production app server: dev/tests parse the live source, Trigger.dev workers (Node, no bundle shipped) stay in-process
- Bundle:
build:xlsx-worker(bun build --target=node, builtins only, ~0.8 MB) runs inbun run build; shipped via DockerfileCOPY(fails the image build if missing) plus anoutputFileTracingIncludesentry, resolved from cwd candidates like the isolated-vm worker - Bun ignores
resourceLimits, so the worker heap counts against the process exactly like an in-process parse; the bound is concurrency, not heap size - PDF is left in-process: #9311 time-slices its loops, and pdf.js needs the native canvas binding inside the worker
Type of Change
- Improvement
Testing
xlsx-worker-pool.test.tsbuilds the real bundle withbun run build:xlsx-workerand checks in-process parity (preview, complete, text-budget error, archive-guard rejection, corrupt ZIP), caller buffer intact, fallback for a missing bundle / a bundle that throws on load / a worker that exits mid-parse, abort of a running job (frees the slot of a spinning worker) and of a queued job. Each guard was reverted and its test went red- Bun 1.4.2 event-loop gap (max timer gap during one parse):
| Workbook | Mode | In-process | Worker | Output hash |
|---|---|---|---|---|
| 16 MB | preview | 1112 ms | 10 ms | identical |
| 16 MB | complete | 1380 ms | 8 ms | identical |
| 23 MB | preview | 1525 ms | 10 ms | identical |
| 23 MB | complete | 1934 ms | 14 ms | identical |
- End to end through
parseBufferwithNODE_ENV=production: worker 66 ms gap (cold spawn included); bundle removed → one warn, in-process, same hash; bundle replaced with a throwing file → one warn, in-process, same hash lib/file-parsers/suite (316 tests),bun run lint, focused tsc on the new modules and test,check:source-conventions,check:utils,check:comment-hygiene,check:file-names,check:test-patterns,check:boundaries
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing (new tests pass the
test-auditauthoring gate) - No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)