Information Extraction Fusion

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• am_diag/graph_construction/extract/llm.py
• am_diag/graph_construction/normalize.py
• am_diag/ingestion/extraction_pipeline.py
• am_diag/ingestion/corpus_ingestion.py
• am_diag/llm/baml_client/inlinedbaml.py

Information Extraction Fusion is a core architectural pattern in the am_diag system designed to synthesize structured clinical knowledge from unstructured medical literature. It employs a "three-extractor fusion" strategy that combines local zero-shot models with Large Language Models (LLMs) to populate a Neo4j knowledge graph. This multi-modal approach ensures high recall and precision by merging different extraction signals into a unified set of entities and relations.

Sources: README.md:19-21, am_diag/ingestion/corpus_ingestion.py:108-112

Extraction Architecture

The fusion process begins by processing document chunks through three specialized extractors. These extractors are driven by a runtime-injectable schema that defines medical entity types (e.g., Disease, Drug) and clinical relationship types (e.g., TREATED_BY, CONTRAINDICATED_IN).

Extractor Components

Extractor	Technology	Primary Role
GLiNER	Zero-shot NER	Performs batch entity extraction based on natural-language labels.
GLiREL	Zero-shot RE	Extracts relations between the spans identified by GLiNER.
LLM	BAML / OpenAI / Anthropic	Uses dynamic enums and structured prompting for high-reasoning extraction.

Sources: README.md:19-26, am_diag/graph_construction/extract/llm.py:84-88

LLM Extraction Logic

The LLM extractor utilizes BAML (Boundary ML) to enforce schema constraints. It converts the graph schema into dynamic enums at runtime, ensuring the model only returns valid entity and relation types. The extraction process is concurrent, utilizing semaphores to manage llm_max_concurrent limits.

The flattening process validates that head_name and tail_name of every relation exist within the extracted entities list for that chunk to ensure graph integrity.
Sources: am_diag/graph_construction/extract/llm.py:44-82, am_diag/llm/baml_client/inlinedbaml.py:171-205

Fusion and Normalization Pipeline

Once raw data is gathered from the extractors, the system executes a multi-stage fusion and normalization pipeline orchestrated via a LangGraph StateGraph.

Fusion Strategies

The system supports several configurable strategies for combining results:

• Union: Merges all unique items, combining provenance records.
• Intersection: Retains only items identified by all active extractors.
• Max Score: Selects the version of an entity or relation with the highest confidence score.
• GLiNER Primary: Uses GLiNER's spans as the anchor, supplemented by LLM findings.

Sources: README.md:112-117, am_diag/ingestion/extraction_pipeline.py:46-51

Data Flow Diagram

The following diagram illustrates the lifecycle of a medical fact from raw text to the normalized state stored in the knowledge graph.

Per-Chunk Normalization

The EntityRelationNormalizer is responsible for cleaning raw extraction outputs before they are globally aggregated. It uses a (normalized name, label) key for entities and a (normalized head, normalized tail, type) key for relations.

Key Logic Steps:

1. Filtering: Removes entities that fall below min_entity_score or min_entity_length.
2. Canonicalization: Uses normalize_text to strip casing and whitespace for key matching while preserving the "best" human-readable name based on the highest score.
3. Provenance Merging: Accumulates chunk_ids, sources, and surface_forms (original text spans) into the Provenance data model.
4. Endpoint Resolution: Relations are updated to point to the canonical names of entities found within the same chunk.

Sources: am_diag/graph_construction/normalize.py:22-68, am_diag/ingestion/extraction_pipeline.py:53-58

Schema Constraints in Fusion

The fusion process is strictly governed by the MEDICAL_GRAPHRAG_SCHEMA. This schema defines 13 entity types and 25 relation types, providing natural-language hints to the zero-shot extractors and JSON definitions to the LLM.

Relationship Group	Example Relations	Head Type -> Tail Type Constraint
Clinical	TREATED_BY	Disease -> {Drug, Procedure}
Pharmacological	TARGETS	Drug -> {Protein, Gene}
Molecular	ENCODES	Gene -> Protein

Sources: README.md:70-98, am_diag/graph_construction/extract/llm.py:47-50

Conclusion

Information Extraction Fusion provides the necessary structure for the Agentic Med-Diag system by converting raw text into high-fidelity knowledge. By combining the speed of local models with the reasoning depth of LLMs and enforcing strict schema validation, the system creates a reliable foundation for subsequent community detection and diagnostic reasoning tasks.

Sources: README.md:123-125, am_diag/ingestion/corpus_ingestion.py:165-175