fix(rag): preserve JSON format metadata in JsonChunker

JU
1d ago
+14-1

Summary

JsonChunker first tokenizes its flattened JSON representation, then wraps the token chunks with JSON-specific metadata. The inherited token metadata contains format: "text", and buildMetadata merges the inherited extras last, overwriting the intended format: "json". Valid JSON chunks therefore silently report the wrong format even though sourceType is JSON and structural fields are present.

This PR removes the inherited format and sourceType keys before merging token chunk metadata. It retains token positions and JSON structural metadata, while invalid JSON still falls back to token chunking with format: "text".

Changes

  • Exclude the inner TokenChunker's format and sourceType from JsonChunker's extras.
  • Assert JSON format/source type on parsed JSON and text/token metadata on the invalid-JSON fallback.
  • Add a patch Changeset for @voltagent/rag.

Reproduction and validation

  • Examined the current upstream main source at 72a46c76b4507e1f98b41d5050a92bd30587eb70.
  • Isolated TypeScript harness recreated the reviewed JsonChunker, TokenChunker, and buildMetadata implementations, substituting only minimal tokenization/position helpers because the complete monorepo dependencies were unavailable.
  • Before fix: valid JSON produces format: "text"; the targeted assertion fails.
  • After fix: TypeScript compilation and targeted assertions pass for valid JSON (json/json) and invalid JSON fallback (text/token).
  • Patch verified to apply cleanly to the reviewed source snapshot and passed git diff --check.

Not run: official monorepo Vitest, package TypeScript build, lint, full CI, and integration tests. These still need real execution in the upstream/fork CI environment. The isolated harness is not a substitute for official tests.

Scope and prior work

Independent of #1459, which changes detectFormat routing for top-level arrays; this PR fixes metadata produced after a valid JSON payload has already reached JsonChunker.

AI use

Prepared with AI-assisted source review, duplication search, isolated reproduction, and patch verification. All validation limits are documented above.

Summary by cubic

Fixes JsonChunker metadata so valid JSON chunks report format: "json" and sourceType: "json" instead of inheriting "text"/"token" from the inner token chunker.

  • Strips the token chunk's format and sourceType before merging JSON metadata, while keeping token positions and structural fields.
  • Adds assertions covering the JSON metadata path and the invalid-JSON fallback, plus a patch Changeset for @voltagent/rag.

Summary by CodeRabbit

  • Bug Fixes
    • JSON chunks now preserve their format and source type metadata, including when the input contains invalid JSON. This keeps information about the original content available alongside chunks produced by the JSON chunker.