Text Chunking Strategies
Relevant source files
The following files were used as context for generating this wiki page:
Text chunking serves as the bridge between raw document ingestion and downstream processing in the agentic-med-diag project. Its primary purpose is to decompose large medical documents into manageable Chunk DataPoints, which are then used for knowledge graph extraction, embedding, and vector search. This process ensures that context remains coherent while meeting the token limits of Large Language Models (LLMs) used in the extraction and retrieval stages. Sources: README.md:104-106, am_diag/chunking/__init__.py
The system primarily utilizes a recursive character splitting strategy, supplemented by specialized loaders for structured content. Each chunk maintains deterministic identity and full provenance back to its source document, allowing for accurate citation in the final clinical reasoning loop. Sources: am_diag/chunking/recursive_character_text_splitter.py:11-13, tests/am_diag/chunking/test_recursive_character_text_splitter.py:84-93
Architecture and Data Flow
The chunking module sits within the broader ingestion pipeline. After documents are streamed from corpus loaders (such as StatPearls or PubMed), they are passed to a chunker which transforms a list of Document objects into a flat list of Chunk objects.
The diagram shows how the chunking strategy serves as a pre-requisite for both graph construction and vector storage. Sources: README.md:112-115, am_diag/chunking/recursive_character_text_splitter.py:65-72
Recursive Character Text Chunker
The RecursiveCharacterTextChunker is the core implementation for splitting medical text. It leverages langchain-text-splitters to recursively split text using a hierarchy of separators to keep related text (like paragraphs and sentences) together as much as possible. Sources: am_diag/chunking/recursive_character_text_splitter.py:8-13, pyproject.toml:44
Splitting Logic
The chunker attempts to split text using an ordered list of separators. If a split results in a segment larger than the chunk_size, it moves to the next separator in the list. By default, it uses ["\n\n", "\n", " ", ""]. Sources: am_diag/chunking/recursive_character_text_splitter.py:22-24
Key features of this strategy include:
- Deterministic Identity: Chunks can generate stable UUID5 identifiers based on the
document_idandchunk_index, ensuring consistent processing across pipeline runs. Sources: tests/am_diag/chunking/test_recursive_character_text_splitter.py:84-93 - Provenance Tracking: Each chunk carries its
document_id,document_source, andchunk_indexto maintain traceability. Sources: am_diag/chunking/recursive_character_text_splitter.py:91-96 - Whitespace Management: Leading and trailing whitespace is stripped by default to clean the input for LLM extractors. Sources: am_diag/chunking/recursive_character_text_splitter.py:34-35
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
chunk_size | int | 4000 | Maximum character count per chunk. |
chunk_overlap | int | 200 | Overlap in characters between consecutive chunks. |
separators | list[str] | None | Hierarchy of split characters (Defaults to newline/space). |
strip_whitespace | bool | True | Removes surrounding whitespace from chunks. |
add_start_index | bool | False | Includes character offset from source in metadata. |
Sources: am_diag/chunking/recursive_character_text_splitter.py:38-51
Chunk Data Model
The Chunk is a specialized DataPoint designed to hold both the textual content and the metadata required for graph and vector operations. Sources: tests/am_diag/common/data_models/test_chunk.py:10-12
The relationship shows that a single Document is decomposed into multiple indexed Chunks. Sources: am_diag/chunking/recursive_character_text_splitter.py:80-104
Identity and Determinism
A critical aspect of the chunking strategy is the deterministic generation of chunk IDs. When a document_id and chunk_index are provided, the system can produce the same UUID for the same segment of text across different sessions. This is vital for deduplication in the knowledge graph. Sources: tests/am_diag/common/data_models/test_chunk.py:46-75
# From tests/am_diag/chunking/test_recursive_character_text_splitter.py:84-93
async def test_deterministic_uuid5(self):
doc = _make_doc(text="Same content.", external_id="stable-doc")
c1 = (await RecursiveCharacterTextChunker(chunk_size=100).chunk([doc]))[0]
doc2 = _make_doc(text="Same content.", external_id="stable-doc")
c2 = (await RecursiveCharacterTextChunker(chunk_size=100).chunk([doc2]))[0]
assert c1.id == c2.id
Specialized Loaders
In addition to character-based splitting, the system integrates MarkItDownLoader (via the markitdown package). This suggests a strategy for handling structured document formats (like PDF, DOCX, or Excel) by first converting them to Markdown before performing chunking. This preserves structural context like tables and headers that might be lost in raw text extraction. Sources: am_diag/chunking/__init__.py:3-4, pyproject.toml:45
Summary
The text chunking strategies in agentic-med-diag prioritize context preservation and traceability. By utilizing a recursive splitting mechanism with configurable overlaps and separators, the system ensures that medical concepts are not sliced mid-sentence. The deterministic ID generation and robust Chunk data model provide the foundation for a reliable knowledge graph where clinical evidence can be traced back to its specific source passage. Sources: README.md:120-125, am_diag/chunking/recursive_character_text_splitter.py:11-15