Corpus Ingestion Loaders
Relevant source files
The following files were used as context for generating this wiki page:
Corpus Ingestion Loaders are the entry point for the medical knowledge graph construction pipeline. They are responsible for streaming raw medical literature from external sources—primarily HuggingFace datasets—and transforming them into a standardized Document format that the downstream Ingestion Pipeline can process.
These loaders use asynchronous iterators to stream data in batches, ensuring that large-scale medical corpora like PubMed or StatPearls can be ingested without exhausting system memory. Each loader implementation encapsulates the logic for mapping source-specific metadata (e.g., article IDs, section types) into a common property schema used for tracking provenance and relevance during extraction.
Sources: README.md:129-131, am_diag/ingestion/corpus_ingestion.py:100-110, CONTRIBUTING.md:46-47
Architecture and Data Flow
The ingestion process follows a strictly defined flow where loaders act as the primary data producer. The run_all_corpus_ingestion orchestrator iterates through a registry of loaders, triggering a sequential stream of document batches.
Ingestion Flow Diagram
The following diagram illustrates how raw data moves from HuggingFace through the loaders and into the core ingestion logic.
The Corpus Loader Registry
Loaders are managed via the CORPUS_LOADER_REGISTRY, which maps configuration names to specific loader classes. This allows the system to resolve which datasets to ingest based on CLI arguments or environment settings.
| Corpus Name | Loader Class | Description |
|---|---|---|
clinical_guidelines | ClinicalGuidelinesCorpusLoader | Authoritative health organization guidelines. |
pubmed | PubMedCorpusLoader | Biomedical titles and abstracts. |
pubmed_case_reports | PubmedCaseReportsCorpusLoader | Full-text clinical presentations and outcomes. |
statpearls | StatPearlsCorpusLoader | Peer-reviewed clinical reference articles. |
textbooks | TextbooksCorpusLoader | English USMLE preparation textbooks. |
Sources: am_diag/pipelines/ingestion.py:44-51, README.md:278-285
Implementing a Loader
New loaders must subclass the CorpusLoader base class and implement the _row_to_document method. This method acts as a parser for individual records within the HuggingFace dataset.
Core Components
corpus_name: A unique identifier for the corpus.hf_repo: The HuggingFace repository path._row_to_document(row): Logic to transform a dictionary (row) into aDocumentobject.astream(): An inherited method that handles asynchronous batching and threading for HuggingFace iterators.
Sources: CONTRIBUTING.md:46-49, am_diag/loaders/corpus/statpearls.py:12-16
Case Study: StatPearls Loader
The StatPearlsCorpusLoader demonstrates how source-specific IDs and titles are parsed into structured metadata. It extracts the article_id and chunk_index from the row ID (e.g., article-100024_2) and identifies the section_type from the title string (e.g., Chronic Occlusion -- Introduction).
# am_diag/loaders/corpus/statpearls.py:32-52
def _row_to_document(self, row: dict[str, Any]) -> Document | None:
text = row.get("content")
if not text:
return None
doc_id = row["id"]
stripped = doc_id.removeprefix("article-")
article_id, _, index_str = stripped.rpartition("_")
# ... logic to determine chunk_index and section_type ...
return Document(
text=text,
source="statpearls",
external_id=doc_id,
title=title,
properties={
"article_id": article_id,
"section_type": section_type,
"chunk_index": chunk_index,
},
)
Sources: am_diag/loaders/corpus/statpearls.py:12-65, tests/am_diag/loaders/corpus/test_statpearls_row_to_document.py:15-55
Batch Processing and Orchestration
The run_corpus_ingestion function consumes the loader's stream. It processes documents in batches, typically sized at 100 documents by default.
Execution Sequence Diagram
The sequence below depicts the interaction between the loader and the ingestion orchestrator during a single batch cycle.
Configuration and CLI Usage
Loaders are typically invoked via the am-diag-ingest CLI tool. Users can specify specific corpora or ingest the entire registry.
- Command:
uv run am-diag-ingest --corpus <names> --batch-size <n> - Arguments:
--corpus: Comma-separated list of loaders (e.g.,pubmed,statpearls) orall.--batch-size: The number of documents to stream per iteration.
Sources: am_diag/pipelines/ingestion.py:110-145, pyproject.toml:47
Conclusion
Corpus Ingestion Loaders provide a standardized interface for medical data acquisition. By abstracting the complexities of HuggingFace streaming and source-specific parsing into a registry of modular classes, the system ensures that the knowledge graph can be easily extended with new medical literature while maintaining rigorous provenance tracking for the clinical diagnosis agent.