improvement(file-parsers): parse spreadsheets on a worker thread

Merged
WA
19d ago
+681-7

Summary

  • XLSX.read is synchronous, so every spreadsheet parse through parseBuffer blocked the event loop for the whole parse (1.1–2s on 16–23 MB workbooks on Bun)
  • Spreadsheet buffers (xlsx/xls/xlsm/xlsb/xltx/ods) now parse on a worker_threads pool running the same XlsxParser; output is identical
  • Pool is bounded to one worker (in-process parses were already serialized by the sync parse, so peak workbook memory stays where it is); jobs queue, idle workers terminate after 30s
  • Input bytes are copied into a fresh ArrayBuffer and transferred, so the caller's Buffer is never detached; options cross without the signal
  • Abort: a queued job is dropped, a running job's worker is terminated and the call rejects with signal.reason (same value the in-process path throws)
  • Errors rebuild as the same class (FileParserError with code/message/cause name+message, ZipBombError, ArchiveIntegrityError), so getFileParserErrorCode callers see no change
  • Safe fallback: missing bundle, load failure, startup that never reports ready, or a crash mid-parse disables the pool for the process (one warn) and runs the job in-process exactly as before
  • Enabled only for the production app server: dev/tests parse the live source, Trigger.dev workers (Node, no bundle shipped) stay in-process
  • Bundle: build:xlsx-worker (bun build --target=node, builtins only, ~0.8 MB) runs in bun run build; shipped via Dockerfile COPY (fails the image build if missing) plus an outputFileTracingIncludes entry, resolved from cwd candidates like the isolated-vm worker
  • Bun ignores resourceLimits, so the worker heap counts against the process exactly like an in-process parse; the bound is concurrency, not heap size
  • PDF is left in-process: #9311 time-slices its loops, and pdf.js needs the native canvas binding inside the worker

Type of Change

  • Improvement

Testing

  • xlsx-worker-pool.test.ts builds the real bundle with bun run build:xlsx-worker and checks in-process parity (preview, complete, text-budget error, archive-guard rejection, corrupt ZIP), caller buffer intact, fallback for a missing bundle / a bundle that throws on load / a worker that exits mid-parse, abort of a running job (frees the slot of a spinning worker) and of a queued job. Each guard was reverted and its test went red
  • Bun 1.4.2 event-loop gap (max timer gap during one parse):
WorkbookModeIn-processWorkerOutput hash
16 MBpreview1112 ms10 msidentical
16 MBcomplete1380 ms8 msidentical
23 MBpreview1525 ms10 msidentical
23 MBcomplete1934 ms14 msidentical
  • End to end through parseBuffer with NODE_ENV=production: worker 66 ms gap (cold spawn included); bundle removed → one warn, in-process, same hash; bundle replaced with a throwing file → one warn, in-process, same hash
  • lib/file-parsers/ suite (316 tests), bun run lint, focused tsc on the new modules and test, check:source-conventions, check:utils, check:comment-hygiene, check:file-names, check:test-patterns, check:boundaries

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)