fix(rag): keep NeuralChunker output IDs distinct after expansion
PR Checklist
- Conventional commit message.
Bugs / Features
- Related issue and work plan filed before implementation: #1518.
- Regression tests added using the real default and whitespace tokenizers.
- Docs updated: N/A, restores distinct per-call chunk IDs without adding an API.
- Patch changeset for
@voltagent/rag.
What is the current behavior?
A detector segment after token fallback keeps its pre-expansion raw index. For "a b c d e f", detector offsets [1, 9], and budget 2, NeuralChunker returns IDs neural-0, neural-1, neural-2, neural-2. The last two chunks contain different text. Writing that one call's four chunks to the actual InMemoryVectorAdapter by ID leaves only three records. #1518 contains the public reproduction on RAG 1.0.2 / main e9fc599.
What is the new behavior?
Assign a fitting detector segment's ID from its final append index, matching the existing token-fallback path. This is one production line. Chunk IDs are distinct in output order within a call, under the existing label prefix. It does not add cross-document/global ID guarantees or change splitting, contents, ranges, token counts or metadata.
Fixes #1518.
Notes for reviewers
- Focused regressions on the original implementation: 2 failed / 3 existing tests passed. The two parameterized cases cover the actual default tiktoken and whitespace tokenizers, each with normal/custom labels after the fix.
- Full RAG suite after the fix: 49 passed in 20 files. New regressions are colocated in
src/chunkers/neural-chunker.spec.ts; the originalsrc/__tests__/neural-chunker.spec.tsis preserved. - Independent built-public-entrypoint probe: four collision cases before → 4/4 passes after. The real in-memory vector store changes from 3 to 4 records in every case, and each chunk is retrieved with its own text. The probe also compares all chunk fields except IDs before/after and confirms they are unchanged.
- RAG CJS/ESM/declaration build, full
pnpm lint:ci(five existing warnings), targeted Biome checks, andgit diff --checkpass. Normal pre-commit and commit-msg hooks ran successfully. - Standalone RAG
tsc --noEmitreports three existing test diagnostics:code-chunker.spec.ts:60andlate-chunker.spec.ts:16,27. Running with the exact base production source and base neural test reproduces the identical diagnostics. No new diagnostic is introduced; this check is not reported as passing.
All 104 open PR heads were checked against the full prior file audit. Existing RecursiveChunker sentence-expansion and shared-metadata fixes leave this NeuralChunker path unchanged. This branch is based directly on upstream and fixes only the linked NeuralChunker issue.
Node 24.13.0/macOS arm64. The detector is an explicitly configured deterministic boundary callback, as supported by the API; no neural model runs. The vector-store probe supplies valid manual [1] vectors and scalar text metadata, with no embedding/model substitute presented as a real model. Actual tokenizers, chunker and vector adapter run without mocks. Full monorepo tests/build, provider/cloud integrations, browser and OS/Node matrices were not run.
AI assistance: OpenAI Codex discovered, implemented and locally reviewed this change, and ran the checks above under the contributor's authorization. Local automated verification does not represent human or maintainer approval.
Summary by cubic
Fixes NeuralChunker returning duplicate chunk IDs after token expansion, which caused distinct chunks to overwrite each other in ID-keyed stores like InMemoryVectorAdapter.
Previously a detector segment that fell back to token chunking kept its pre-expansion raw index — "a b c d e f" with budget 2 produced neural-0, neural-1, neural-2, neural-2. Now each chunk's ID comes from its final output position, matching the existing token-fallback path. Splitting, contents, ranges, token counts, and metadata are unchanged, and no cross-document ID guarantees are added.
Tests
- Regression coverage for the default tiktoken and whitespace tokenizers, with normal and custom labels.
This fixes #1518.
Summary by CodeRabbit
- Bug Fixes
- Fixed chunk identifiers so token-limited neural chunks receive distinct, sequential IDs, including when detected segments are split into multiple chunks.
- Ensured this behavior is consistent with the default and whitespace tokenizers, for both neural and custom labels.