Hierarchical Community Detection
Relevant source files
The following files were used as context for generating this wiki page:
- README.md
- am_diag/graph_construction/community/base.py
- am_diag/graph_construction/community/leiden.py
- am_diag/graph_construction/community/gds.py
- am_diag/graph_construction/community/summarize.py
- am_diag/graph_construction/community/context.py
- am_diag/common/data_models/community.py
- am_diag/common/data_models/community_report.py
Hierarchical Community Detection is a core component of the am_diag ingestion pipeline that partitions a resolved medical knowledge graph into nested clusters of related entities and relationships. By applying the Leiden algorithm, the system organizes clinical knowledge into levels of granularity, ranging from broad medical domains to specific physiological mechanisms. This structure allows the Agentic RAG system to navigate the graph at different scales, enabling thematic retrieval through community-level summaries.
Sources: README.md:14-17, README.md:126-131
Detection Process and Backends
The system supports two interchangeable backends for community detection, both inheriting from a common base class. Detection occurs after entity resolution to ensure clusters are formed over canonical entities rather than surface-form variants.
Backend Architectures
- Graspologic (Local): Uses the
graspologic-nativelibrary to run hierarchical Leiden clustering in-memory. It constructs an edge list from the input relations and returns a hierarchy of communities. - Neo4j GDS (Remote): Uses the Neo4j Graph Data Science (GDS) library. It projects the existing graph into a GDS named graph, executes the Leiden algorithm via procedures, and reads the hierarchical assignments back into the application.
Sources: am_diag/graph_construction/community/leiden.py:16-24, am_diag/graph_construction/community/gds.py:11-20, README.md:149-152
Detection Logic Flow
The following diagram illustrates the flow from resolved relations to hierarchical community data points.
The system transforms flat relations into a structured hierarchy of Community objects.
Sources: am_diag/graph_construction/community/leiden.py:35-71, am_diag/graph_construction/community/gds.py:100-142
Community Summarization
Once the hierarchy is established, the system generates natural language reports for every community. This process is performed bottom-up, allowing higher-level reports to incorporate summaries from their child communities.
Report Generation Workflow
The CommunitySummarizer orchestrates the following steps for each level:
- Context Building: For a given community, a text context is generated containing its entities, relationships, and summaries of its immediate sub-communities.
- LLM Generation: The context is passed to a BAML-defined LLM function (
GenerateCommunityReport) along with the graph schema to produce a structured report. - Data Persistence: The output is mapped to a
CommunityReportDataPoint, which includes clinical findings and importance ratings.
Sources: am_diag/graph_construction/community/summarize.py:65-103, am_diag/graph_construction/community/context.py:11-30
Summary Data Flow
Higher-level summaries are enriched by the findings of lower-level communities.
Sources: am_diag/graph_construction/community/summarize.py:84-110, am_diag/graph_construction/community/context.py:46-56
Data Models and Configuration
The detection and summarization modules rely on specific data structures and configuration parameters to control the granularity and quality of the generated hierarchy.
Community Detection Configuration
| Parameter | Type | Default | Description |
|---|---|---|---|
max_cluster_size | int | 1000 | Maximum number of nodes allowed in a single community. |
resolution | float | 1.0 | Controls cluster granularity (higher = smaller clusters). |
seed | int | 42 | Random seed for deterministic clustering. |
iterations | int | 10 | Number of Leiden refinement iterations. |
Sources: am_diag/graph_construction/community/leiden.py:28-29, am_diag/graph_construction/community/gds.py:35-36
Community Report Structure
The CommunityReport model captures the medical intelligence extracted from a cluster.
| Field | Type | Description |
|---|---|---|
community_id | UUID | Reference to the Community DataPoint. |
title | str | Short descriptive title for the community. |
summary | str | 2-3 sentence overview of medical significance. |
findings | list[Finding] | Structured list of headlines and evidence. |
rating | float | Importance rating (0-10) for clinical decision-making. |
Sources: am_diag/common/data_models/community_report.py:23-33, am_diag/llm/baml_client/inlinedbaml.py: community.baml
Clinical Significance
Hierarchical community detection is vital for the Agentic RAG loop. It allows the Researcher agent to perform "thematic retrieval." Instead of retrieving thousands of individual atomic facts for a broad question like "What are the common comorbidities of Diabetes?", the agent can retrieve and read a single CommunityReport for a high-level community focused on Metabolic Diseases. This significantly reduces the token pressure on the LLM and improves the synthesis of complex medical answers.
Sources: README.md:129-131, README.md:214-220