Testing Strategy & Test Suite
Relevant source files
The following files were used as context for generating this wiki page:
The am_diag project employs a multi-layered testing strategy designed to ensure the reliability of its agentic medical diagnosis pipelines. The suite is partitioned into unit tests, which are isolated from the network to verify individual component logic, and integration tests that validate end-to-end flows involving external services like Neo4j, Qdrant, and LLM endpoints.
The testing infrastructure is built on pytest and leverages pytest-asyncio for asynchronous code execution, pytest-socket for network isolation, and pytest-xdist for parallel execution. This structure ensures that both core data models and complex ingestion/retrieval workflows remain robust as the system evolves.
Sources: CONTRIBUTING.md:104-110, pyproject.toml:57-67
Test Architecture and Organization
The test suite mirrors the structure of the am_diag/ package, located under the tests/am_diag/ directory. This parallel structure allows developers to easily locate tests corresponding to specific modules, such as loaders, ingestion, or common utilities.
The diagram shows the hierarchical organization of the test suite mirroring the application package.
Sources: CONTRIBUTING.md:46-57, CONTRIBUTING.md:104-105
Test Categories
The project distinguishes between tests based on their reliance on external dependencies:
| Category | Description | Command |
|---|---|---|
| Unit Tests | Mocks external services (Neo4j, Qdrant, LLMs). Network access is disabled via pytest-socket. | make test |
| Integration Tests | Requires active external services. Marked with @pytest.mark.integration. | make test-integration |
| Coverage | Generates reports on code paths executed during testing. | make test-cov |
Sources: CONTRIBUTING.md:107-111, CONTRIBUTING.md:120-125
Unit Testing Strategy
Unit tests focus on the deterministic behavior of components. For instance, corpus and dataset loader tests verify that raw data rows are correctly transformed into internal data models while handling edge cases like empty questions or invalid answer indices.
Dataset Loader Validation
Tests for loaders like MedMCQADataset and PubMedQADataset enforce strict validation rules. If a row lacks a question or contains an invalid answer key (e.g., cop=0), the loader must return None or skip the row to maintain data integrity.
def test_skips_row_with_empty_question(self, patch_load_dataset):
patch_load_dataset(
"am_diag.loaders.dataset.base",
[make_pattern_b_medmcqa_row(question="")],
)
samples = MedMCQADataset().load()
assert len(samples) == 0
Sources: tests/am_diag/loaders/dataset/test_medmcqa.py:53-59, tests/am_diag/loaders/dataset/test_pubmedqa.py:46-52
Cypher Query Validation
The am_diag/common/cypher tests ensure that Cypher query templates are safely rendered. A safe_identifier utility validates node labels and relationship types to prevent injection attacks before they are sent to the Neo4j client.
Sources: tests/am_diag/common/cypher/test_loader.py:11-30
Integration Testing Strategy
Integration tests validate the interaction between the am_diag code and real-world services. These tests are essential for verifying the streaming capabilities of HuggingFace dataset loaders and the correctness of complex graph queries.
Sequence of an integration test verifying data streaming and database persistence.
Sources: tests/am_diag/loaders/dataset/test_pubmedqa_integration.py:31-45, CONTRIBUTING.md:109-111
Data Invariants in Integration
Integration tests for datasets (e.g., PubMedQA) assert specific invariants on real data, such as:
- Ensuring every sample has at least 3 options.
- Verifying that question stems contain necessary context labels.
- Checking that sample IDs are unique and correctly derived from source IDs (e.g., PubMed IDs).
Sources: tests/am_diag/loaders/dataset/test_pubmedqa_integration.py:47-58, tests/am_diag/loaders/dataset/test_pubmedqa_integration.py:100-104
Test Configuration and Fixtures
The project utilizes pytest fixtures extensively to provide reusable, mock data for ingestion and graph construction tests. Fixtures are defined in conftest.py files at various levels of the test directory.
Key Fixtures
| Fixture Name | Description | File Path |
|---|---|---|
sample_entities | Provides a list of mock Entity objects with provenance. | tests/am_diag/ingestion/conftest.py |
sample_relations | Provides mock Relation objects connecting entities. | tests/am_diag/ingestion/conftest.py |
sample_chunks | Mock Document objects representing text segments. | tests/am_diag/ingestion/conftest.py |
minimal_config | An ExtractionPipelineConfig with all extractors disabled for fast testing. | tests/am_diag/ingestion/conftest.py |
Sources: tests/am_diag/ingestion/conftest.py:16-95
Automated Quality Gates
The project defines several make commands to enforce code quality before submission. These include linting, type checking, and security audits.
make lint-fmt # Format code with Ruff
make lint-typing # Type check with ty
make lint-all # Run all linters (Ruff, ty, typos)
make test-cov # Run tests and generate coverage report
Sources: CONTRIBUTING.md:126-135, pyproject.toml:74-88
The testing strategy ensures that every component, from the BAML-based extraction functions to the recursive-character chunkers, is validated both in isolation and as part of the broader agentic pipeline. By enforcing network isolation in unit tests and providing a dedicated integration suite, the project maintains high velocity without sacrificing medical data accuracy.