Vector Pipeline & Representation
Relevant source files
The following files were used as context for generating this wiki page:
The Vector Pipeline & Representation system is responsible for transforming extracted clinical knowledge and medical literature into high-dimensional vector embeddings for semantic search. This system operates as a secondary stage following graph construction in the ingestion workflow, populating specialized vector collections that complement the Neo4j knowledge graph.
By representating entities, relations, chunks, and community reports as vectors, the system enables the Retrieval Pipeline to perform hybrid search (combining dense vector search with BM25) and multi-strategy retrieval during the agentic reasoning loop.
Sources: README.md:58-62, README.md:144-147, am_diag/ingestion/corpus_ingestion.py:27-29
Pipeline Architecture
The embedding pipeline is orchestrated within the run_corpus_ingestion function. After documents are chunked and clinical elements are extracted and resolved, the system fouts to the vectorisation stage.
Ingestion Flow
- Extraction & Resolution: Entities and relations are extracted from text chunks and canonicalized.
- Vectorization: Textual representations of clinical data models are passed to an
Embedder. - Upsertion: Resulting vectors are packaged into
VectorRecordobjects and sent to aVectorStoreBase(Qdrant or Weaviate).
Data Representation
The system utilizes four distinct vector collections to capture different semantic granularities of the medical corpus.
Vector Collections Summary
| Collection | Embedded Text Source | Purpose |
|---|---|---|
| Entity | Canonical name (or extracted name) | Semantic lookup of medical concepts (Drugs, Diseases). |
| Relation | Triple-based representations (Fact, Type, SPO) | Captures clinical associations and predicates. |
| Chunk | Raw passage text (limited to 500 chars in payload) | Evidence-based retrieval of source literature. |
| Community | LLM-generated report summary | Thematic, cluster-level search for broad queries. |
Sources: README.md:144-152, am_diag/ingestion/corpus_ingestion.py:44-90
Multi-Representational Relation Embedding
A single clinical relation is embedded using three different representations to ensure comprehensive retrieval coverage:
- Edge Fact: The description of the relationship (e.g., "metformin treats type 2 diabetes").
- Edge Type: The predicate/relation type alone (e.g., "TREATED_BY").
- Full SPO: The complete subject-predicate-object sentence structure.
Sources: README.md:154-160, am_diag/ingestion/corpus_ingestion.py:59-69
Data Models
The core data structures for the vector system are defined in am_diag.common.data_models.
VectorRecord
The VectorRecord is the standard unit of storage in the vector database. It decouples the mathematical vector from the descriptive metadata (payload).
class VectorRecord(BaseModel):
id: str | int | UUID
vector: list[float]
payload: dict[str, Any] = Field(default_factory=dict)
Sources: am_diag/common/data_models/vector_record.py:10-14, am_diag/common/data_models/init.py:67
Payload Schemas
When upserting, the system attaches specific metadata to each record to facilitate hydration and filtering during retrieval.
- Entity Payload: Includes
id,name,canonical_name, andlabel(e.g., "Disease"). - Relation Payload: Includes
id,head_name,tail_name, and relationtype. - Chunk Payload: Includes
id, the first 500 characters oftext,document_id, anddocument_source.
Sources: am_diag/ingestion/corpus_ingestion.py:47-88
Implementation Details
The vectorization logic is primarily contained in _embed_and_upsert. This function handles batching and conditional execution based on the availability of extracted data.
The system supports lazy collection creation and native vector quantization as part of the VectorStoreBase implementation, allowing for efficient scaling across large medical corpora like PubMed or StatPearls.
Sources: README.md:162-163, am_diag/ingestion/corpus_ingestion.py:116-120
Conclusion
The Vector Pipeline & Representation module provides the semantic foundation for the Agentic Med-Diag system. By converting structured graph elements and unstructured text into high-dimensional vectors across four specialized collections, it enables the agent to navigate clinical knowledge through both exact graph traversals and semantic similarity searches.