Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The testing strategy for monGARS is a comprehensive framework designed to ensure stability, validate research components, and maintain high code quality across the project. It applies to all code within the tests/ directory, encompassing asynchronous fixtures, integration checks, and specialized long-running suites. The strategy is built around pytest for the Python backend and npm for frontend assets, with a total of 631 tests that run in approximately 110 seconds.
The core philosophy emphasizes creating hermetic tests by resetting shared state in fixtures and stubbing heavy dependencies like Torch, spaCy, and HTTP clients to ensure deterministic runs. This approach supports the project's goals of stability assurance, research coverage, and sustainability validation.
Sources: tests/AGENTS.md, README.md:195-197, AGENTS.md:65-67
Tooling and Execution
The project provides a standardized set of tools and commands for running tests, ensuring consistency for all contributors. All tests should be executed from the repository root to ensure correct module path resolution.
This diagram illustrates the typical testing workflow for a developer.
Sources: tests/AGENTS.md:73-77
Pytest Commands
The primary tool for backend testing is pytest. The following commands are recommended for different scenarios:
| Command | Description |
|---|---|
pytest | The default command to run all 631 tests. |
pytest -k <pattern> | For focused runs on a subset of tests matching a pattern. |
pytest --maxfail=1 | For triage, stopping the test run after the first failure. |
pytest --cov=monGARS --cov=modules | To generate a coverage report for the main application and modules. |
Sources: tests/AGENTS.md:73-77
Frontend Testing
For frontend assets written in TypeScript or JavaScript, the command npm run test should be executed to run linting and Jest test suites. This ensures that web assets and SDKs remain functional and adhere to coding standards.
Sources: tests/AGENTS.md:77, AGENTS.md:66
Test Philosophy and Conventions
To maintain a clean and effective test suite, the project enforces several conventions:
- Asynchronous Tests: All asynchronous tests must be annotated with
pytest.mark.asyncio. Manual event loop creation is forbidden. - Fixtures: Tests should rely on shared fixtures defined in
conftest.py. - State Management: Shared state, such as
hippocampuscaches and global singletons, must be reset within fixtures to ensure tests are hermetic and do not influence each other. - Dependency Stubbing: Heavy dependencies, including ML frameworks (Torch, spaCy) and external services (HTTP clients), must be stubbed using fakes or patches to keep test runs fast and deterministic.
Sources: tests/AGENTS.md:58-63, monGARS/AGENTS.md:104-106
Test Suite Mapping
The test suite is structured to mirror the application's modular architecture. Each major component has a corresponding set of tests to validate its functionality. The AGENTS.md files located within each module's directory specify the key test files that should be maintained.
This diagram shows the high-level mapping from application modules to their test suites.
The following table provides a detailed mapping of application components to their specific test suites:
| Component/Module | Key Test Suites |
|---|---|
monGARS (Core Package) | test_conversation_flow.py, test_llm_integration.py, test_peer_communication.py, test_long_haul_validation.py, test_sustainability_dashboard.py, test_research_long_haul_service.py |
monGARS/api | test_api_chat.py, test_api_history.py, test_api_model_management.py, test_api_rag.py, test_ticket_signer.py, test_websocket.py, test_ws_manager.py |
monGARS/core (Cognition) | test_conversation_semantic_context.py, test_llm_integration.py, test_peer_communication.py, test_reinforcement_loop.py, test_long_haul_validation.py, test_sustainability_dashboard.py |
modules (Optional) | test_evolution_engine.py, test_training_pipeline.py, test_mntp_trainer.py, test_run_dolphin_unsloth_workflow.py, test_mlops_training.py |
webapp (Django UI) | test_webapp_chat_services.py, test_webapp_settings.py, test_webapp_database_settings.py, test_websocket.py |
Sources: monGARS/AGENTS.md:108-112, monGARS/api/AGENTS.md:58-61, monGARS/core/AGENTS.md:72-76, modules/AGENTS.md:46-49, configs/agents/agents_config.json:206-207
Specialized and Long-Haul Suites
For significant changes, such as releases or infrastructure modifications, a set of specialized, long-running test suites must be executed. These include:
chaos_test.pyintegration_test.pyself_training_test.py- Reinforcement long-haul suites
These suites are designed to exercise complex interactions and ensure the system's resilience and stability under various conditions.
Sources: tests/AGENTS.md:67-69
Coverage Expectations
The testing strategy mandates comprehensive coverage for all critical paths:
- Success and Failure Paths: Tests must cover both success and failure scenarios for HTTP endpoints, WebSockets, orchestration flows, sustainability gates, and SDK tooling.
- Property-Based Tests: Any property-based tests should be constrained to ensure they run quickly and deterministically.
- Skipped Tests: Any test suites that are skipped must be documented in the pull request notes.
Sources: tests/AGENTS.md:65-71
Test Dataset Generation
The project includes a script, generate_monGARS_llm_dataset.py, to create datasets for testing and training the LLM. This script programmatically generates prompt-response pairs by analyzing the source code, OpenAPI specifications, and predefined architectural scenarios.
The generation process is as follows:
This script can generate examples for several categories, including:
module_reference: Code summaries, dependencies, and testing strategies.architecture: System flows, observability hooks, and resilience risks.operations: Triage and remediation steps for common operational issues.api: Endpoint usage, payload structures, and testing strategies.
The generated datasets are split into training and validation sets and saved as .jsonl files, along with a report summarizing the generation process.