fix(rag): preserve sentence character boundaries

LJ
16h ago
+35-5

PR Checklist

  • The commit message follows the conventional commit guidelines.

Bugs / Features

  • Related issue linked: fixes #1469.
  • Regression tests added.
  • Docs added / updated (not needed: restores existing sentence boundary behavior).
  • Patch Changeset added for @voltagent/rag.

What is the current behavior?

SentenceChunker uses character ranges to describe grouped sentences, but derives content by treating independently encoded prefix token counts as indexes into full-document BPE tokens. These boundaries can differ. For foo.bar baz.qux end. with maxTokens: 3, it returns foo.bar, baz.qu, x end. while the reported ranges select foo., bar baz., qux end.. Overlapping sentence groups and whitespace merged into punctuation tokens reproduce the mismatch too.

What is the new behavior?

Slice the original text using the already-computed first/last sentence character offsets. This preserves complete matched sentences and makes content agree with its reported source range. Sentence grouping, overlap selection, IDs, token accounting and position metadata are unchanged. It also removes the redundant full-document tokenization and prefix encodings used only for reconstructing content.

This is independent of #1466 (recursive IDs) and #1468 (table delegation offsets). The branch starts directly from upstream main and includes neither patch.

Notes for reviewers

Validation on Windows, Node.js 24.19.0, upstream 72a46c76b4507e1f98b41d5050a92bd30587eb70:

  • Unchanged runtime with the new regressions: 3 fail, 3 existing tests pass. Regressions cover adjacent BPE punctuation boundaries, inter-sentence newlines and sentence overlap, using the real default tokenizer.
  • Fixed full RAG suite: 46 tests pass across 18 files, including all existing chunking strategies (vitest run --config packages/rag/vitest.config.mts packages/rag/src/__tests__).
  • Strict TypeScript check of the two changed TypeScript files: passed.
  • Biome check of the two changed TypeScript files: passed.
  • Actual package tsup build: CommonJS, ESM and declaration outputs passed.
  • git diff --check: passed.
  • Package-wide tsc --project packages/rag/tsconfig.json --noEmit still reports the same 3 pre-existing errors in code-chunker.spec.ts:60 and late-chunker.spec.ts:16,27, verified on untouched upstream with the same tooling. Full package type checking is not claimed to pass.

Tests execute actual source and the declared tokenizer dependency, with isolated repo-compatible tooling. No runtime implementation was recreated or patched for testing. Full monorepo tests, live models and vector-store integration were not run; GitHub CI has not been verified.

AI assistance: Codex inspected, authored and locally validated this scoped fix under LJK-ikun's direction. Automated validation is reported separately from human review; independent human testing or approval is not asserted.

Summary by cubic

Fixes SentenceChunker returning chunk content that diverged from the reported sentence ranges when using BPE tokenization. Content is now sliced directly from the computed sentence character offsets, so each chunk's content matches its source range and complete sentences are preserved. Fixes #1469.

  • Removes the redundant full-document tokenization and per-sentence prefix token counting.
  • Adds regression tests for adjacent punctuation, inter-sentence newlines, and overlapping sentences.

Summary by CodeRabbit

  • Bug Fixes
    • Fixed sentence-based chunking so chunk content preserves the original character boundaries when using BPE tokenizers.
    • Improved handling of punctuation-adjacent and newline-separated sentences.
    • Ensured overlapping chunks retain complete sentences and accurate content offsets.