fix(rag): preserve LaTeX chunk source ranges

XY
18d ago
+198-14

PR Checklist

  • The commit message follows the conventional commit guidelines.

Bugs / Features

  • Related issue(s) linked; no existing issue covers this LaTeX range defect.
  • Tests for the changes have been added.
  • Docs have been added / updated; no API or configuration changes.
  • Changesets have been added.

What is the current behavior?

LaTeX section bodies are trimmed before recursive chunking, but the returned ranges are never translated back to the original input. A citation or highlight using source.slice(chunk.start, chunk.end) selects a heading or another section instead of the chunk's content.

For example, both comparisons below return false on current main:

const source = "\\section{Intro}\nFirst body.\n\\subsection{Next}\nSecond body.";
const chunks = new LatexChunker(whitespaceTokenizer).chunk(source);
chunks.map((chunk) => source.slice(chunk.start, chunk.end) === chunk.content);

What is the new behavior?

Keep each trimmed section body's original start, add it to the child ranges, and compute line/column positions against the original document. In the recursive oversized-sentence fallback, tokenize the original sentence range and preserve its offset within the paragraph, including original-document position metadata. The example returns [true, true]. Regression tests cover preamble and unsectioned text, whitespace, starred headings, repeated bodies, CRLF positions and a section expanded into token chunks.

Notes for reviewers

  • Review follow-up: added JSDoc to the LaTeX section splitter and the three affected chunk methods, explaining source spans, overlap, oversized sentences and tokenizer limitations. This commit changes comments only: TypeScript emits identical JavaScript with comments removed for all three files; all 62 RAG tests and changed-file Biome checks pass. The review bot has not yet confirmed its documentation-coverage warning is cleared.

  • All 62 RAG tests pass. Six initial regressions fail on the original source; review follow-up adds a later oversized sentence and direct fallback regressions with both whitespace and tiktoken tokenizers, all three failing before the follow-up fix. Two additional LF/CRLF position regressions fail before the metadata repair and pass afterward.

  • Changed-file TypeScript checks, Biome checks, and ESM/CJS/declaration builds pass.

  • The full-package tsc --noEmit reports three errors in unchanged code-chunker and late-chunker tests. The same errors reproduce on the actual unchanged upstream baseline.

  • This covers section-splitting and recursive token-fallback offset translation. SentenceChunker now derives content, ranges and position metadata directly from the selected source sentence spans. This fixes preceding spaces/tabs, sentences sharing a tokenizer token, and Unicode offsets that can exceed the source when token strings are decoded separately. Four whitespace regressions and four additional no-space/CJK/emoji regressions fail before their corresponding repairs and pass afterward. Sentence grouping, overlap, budgeting and recursive chunk IDs retain their existing behavior. The independent TokenChunker still has a pre-existing Unicode offset limitation in oversized-token fallback; this PR does not claim to fix arbitrary Unicode token fallback.

  • AI assistance: Codex prepared this patch, ran the validation and performed an independent internal AI review at the account owner's request.

Additional external source-range matrix: 792 public LatexChunker calls / 9,462 chunks. All whitespace-tokenizer inputs and default-tokenizer ASCII inputs satisfy source slicing, bounds and independent line/column checks. Default Unicode token fallback still produces 378 range/content violations from the disclosed unchanged tokenizer limitation; these were recorded, not counted as passes. Repeated-whitespace token-budget undercount also reproduces on original upstream source. No strict default-tokenizer budget or arbitrary Unicode fallback guarantee is claimed.

Summary by CodeRabbit

  • Improvements
    • LaTeX and recursive text chunks now retain offsets and line and column positions that correspond to their locations in the original document.
    • Oversized sentence chunks preserve accurate source positions when split further, including with LF and CRLF line endings.
    • Sentence chunks now report source offsets and position metadata that match their content, including when sentences are separated by spaces or tabs.