fix(parsers): structure-preserving knowledge base parsers with before/after benchmark

Merged
WA
2mo ago
+16102-758

Summary

An audit of the knowledge-base file parsers found that content recall was fine but the text reaching the chunker was structurally noisy: PDFs arrived as one line with running headers and footers mid-sentence and words glued across line breaks; Office tables were exploded one cell per line; spreadsheet dates were indexed as Excel serials (including every Google Sheet synced through Drive); legacy .doc/.ppt fell through to a byte scrape that emitted ZIP names and XML; non-UTF-8 text lost its accents silently; a corrupt PDF raised an untyped error. Each finding was verified against the code and open-source precedent (unpdf, pdf.js, Marker, OmniDocBench, olmOCR-bench, MarkItDown, Docling, unstructured, Tika, SheetJS), fixed, benchmarked, then audited again by six independent reviewers whose findings were fixed in turn.

Fixes

  • PDF (pdf-parser.ts, new pdf-lines.ts, pdf-furniture.ts): rebuild lines and paragraphs from item geometry instead of collapsing whitespace (the collapse was copied from unpdf 1.4.0, fixed upstream in 1.7.0); spaces on x-gaps and breaks on backwards moves so Form-XObject and table-cell fusions stop; dehyphenate only when the document shows the joined word; suppress running headers/footers that repeat across pages (frequency + streak rule, first copy kept, table headers exempt via a text-flow test) and page numbers; join pages with a blank line. The page join is linear; heading markers exist but are off by default.
  • Spreadsheets (xlsx-parser.ts, new sheet-display-text.ts, file viewer): display text ($1,250.00, 20%, 2026-03-04, 30:00, TRUE), ISO dates independent of timezone and epoch, elapsed-time formats and time-only cells rendered as Excel does, General numbers at 15 significant digits with integer IDs exact.
  • DOCX (docx-parser.ts, html-parser.ts): mammoth HTML through the shared structured-text walker so headings, lists, tables (nested tables once), footnotes and endnotes survive; the never-read metadata.html is gone.
  • PPTX / ODT / ODP (new ooxml-presentation.ts, odf-text.ts, office-text.ts): JSZip + htmlparser2 walkers that render table rows, skip slide-number/date/header/footer placeholders and notes-page chrome, keep speaker-note bodies and date fields, handle mc:AlternateContent, skip ODT annotations and tracked deletions, cap XML parts at 16 MiB, and detect encrypted packages.
  • Legacy formats (doc-parser.ts, pptx-parser.ts): .doc via word-extractor@1.0.4 (body, headers, footers, footnotes, endnotes, text boxes); Word 6/95 → unsupported_type. .ppt removed from every allowlist and refused with a typed error; the byte-scrape fallbacks are deleted. The Sim file reader, File block and workspace-file "get content" treat degraded output as a parse failure instead of handing it to the model.
  • Input hygiene (utils.ts, sniff.ts, ooxml-encryption.ts, pdfjs-server.ts, text parsers, connectors): BOM → strict UTF-8 → Windows-1252 decoding with a warning (single-pass, bounded); magic-byte sniffing that overrides an obviously wrong extension, refuses RTF and encrypted packages with typed errors, and keeps ambiguous bytes on the declared route; pdf.js InvalidPDFException/PasswordException wrapped as FileParserErrors; HTML mislabelled as .pdf no longer indexed as a text layer; .ppt rows already in a knowledge base fail permanently instead of burning retries.
  • YAML / JSON: multi-document YAML streams (Kubernetes manifests) parse as one item per document; JSON with comments or trailing commas parses leniently with a warning.

Evaluation framework and benchmark

apps/sim/scripts/parser-eval/ holds the plan, a ground-truth corpus generator (pandoc + typst renders with known headers, footers and tables), a fetcher and SHA-256 manifest for a 961-file real-world corpus, a bun harness over the production parseBuffer path, reference extractors, a scorer modelled on olmOCR-bench unit tests, and a before/after comparer with a regression gate (line recall sampled across the whole document, vocabulary recall, noise, glued tokens, count-aware word depletion). BENCHMARK.md is rendered from the comparer's output and records the metric history.

Final run, 961 files, staging vs this branch: PDF line recall 0.990 → 0.989 and vocabulary 0.977 → 0.976 while output goes from one line to real paragraphs and glued tokens fall 3.2 → 1.2 per file; .doc noise 0.75 → 0.003 with recall 0.52 → 0.79; DOCX/ODT vocabulary recall 0.94 → 0.96 / 0.87 → 0.97; text-only spreadsheets byte-identical; latency flat (mean 30 → 31 ms, p99 303 → 329 ms, max 1.37 → 1.10 s). 56 gate flags remain, each traced to a reference artifact or an intended change (table in BENCHMARK.md). Ground-truth corpus: PDF paragraph retention 0.06 → 0.98, Office table adjacency 0.00 → 1.00, typed-cell presence 0.87 → 1.00, robustness 14/14.

Behaviour changes to note

  • .ppt and Word 6/95 files are refused with typed errors everywhere (File block and Chat previously received scraped text for them).
  • Slide-number, date, header and footer placeholders, ODT comments and tracked deletions are not indexed; repeated PDF running headers and footers are indexed once.
  • Corrupt PDFs still route to OCR as before; only the error type changed.
  • Parsed text differs for every re-indexed document, so chunk boundaries move. Connectors hash source revisions, not decoded text, so there is no mass re-sync.
  • Known limitations (documented in code): rotated side footers survive; DOCX header/footer text is not extracted (mammoth, unchanged); CJK PDFs relying on predefined CMaps are unchanged.

Test plan

  • bun run type-check, Biome, and bun run check:audits (all 46 checks, tool-registry baseline re-recorded for the new parser modules) pass.
  • Full apps/sim vitest: 3404 files / 48,060 tests pass. New unit tests cover line joining (including a 50k-line join under 1 s), dehyphenation, furniture detection with repeated table headers, sheet display text under four timezones and both epochs, the OOXML/ODF walkers, encrypted packages, sniffing and decoding, YAML streams, JSONC, and the degraded-output gates.
  • Benchmark reproducible with bench/build.sh, bench-run.ts, bench-compare.py and bench-summary.py (see README.md).

🤖 Generated with Claude Code