Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Serena project employs a multi-faceted testing strategy designed to validate semantic code intelligence across a wide array of programming languages. The strategy focuses on ensuring that the SerenaAgent and the underlying SolidLanguageServer (SolidLSP) integration correctly handle code retrieval, symbol identification, cross-file navigation, and diagnostic reporting.
The test suite is built using pytest and heavily utilizes parameterized testing to run identical semantic checks against different language backends. This ensures consistency in how Serena interacts with Language Server Protocol (LSP) implementations for languages ranging from Python and Java to specialized environments like Zig, Swift, and MATLAB.
Test Categories and Architecture
The testing architecture is divided into specialized modules targeting different layers of the system:
1. Agent Integration Testing
These tests validate the high-level SerenaAgent and its toolset. They verify that the agent can successfully execute tasks such as finding declarations, implementing safe deletes, and managing project-wide configurations.
Key Components:
- SerenaAgent Fixture: Initializes a headless agent with project-specific configurations for different languages.
- Tool Validation: Direct testing of classes like
FindSymbolTool,FindDeclarationTool, andReplaceContentTool.
Sources: test/serena/test_serena_agent.py:463-500, test/serena/test_serena_agent.py:533-550
2. Language-Specific Integration (SolidLSP)
These tests target the SolidLanguageServer directly to ensure correct LSP communication. They utilize "test repositories" (mock codebases) for each supported language.
Core Verification Areas:
- Symbol Retrieval: Validating
request_document_symbolsandrequest_full_symbol_tree. - Navigation: Testing
request_definitionandrequest_references. - Diagnostics: Ensuring
request_text_document_diagnosticscorrectly identifies errors in sample files.
Sources: test/solidlsp/python/test_python_basic.py:18-40, test/solidlsp/zig/test_zig_basic.py:27-40, test/solidlsp/markdown/test_markdown_basic.py:26-45
3. Error Handling and Edge Cases
Specialized tests ensure the system handles invalid states gracefully, such as negative line numbers, non-existent files, or symbols beyond file boundaries.
Sources: test/solidlsp/vue/test_vue_error_cases.py:34-70
Testing Workflow
The following diagram illustrates the typical flow of an integration test where the SerenaAgent utilizes a tool to interact with a Language Server.
The diagram shows the sequence from a test initiation to the external LSP interaction.
Sources: test/serena/test_serena_agent.py:540-560, test/solidlsp/python/test_python_basic.py:30-40
Cross-Language Parameterization
Serena uses a common set of test cases defined as dataclasses (e.g., FindSymbolCase, FindReferenceCase) which are then applied across multiple languages using pytest.mark.parametrize.
| Case Class | Purpose | Data Fields |
|---|---|---|
FindSymbolCase | Validates symbol kind and location. | language, symbol_name, expected_kind, expected_file |
FindReferenceCase | Validates cross-file reference finding. | language, symbol_name, definition_file, reference_file |
DiagnosticCase | Validates error reporting accuracy. | language, relative_path, name_path, message_fragment |
Sources: test/serena/test_serena_agent.py:45-120
Implementation Details
File Modification Context
To test editing tools without corrupting the test environment, the suite uses a context manager that reads the original file content and restores it after the test execution.
@contextmanager
def project_file_modification_context(serena_agent: SerenaAgent, relative_path: str) -> Iterator[None]:
project = serena_agent.get_active_project()
original_content = read_project_file(project, relative_path)
try:
yield
finally:
with open(file_path, "w") as f:
f.write(original_content)
Sources: test/serena/test_serena_agent.py:516-528
Language-Specific Configurations
Different languages have unique constraints handled in the test suite:
- Zig: Requires files to be explicitly "opened" in the virtual editor to find cross-file references due to ZLS optimizations.
- Swift: Skipped on Windows due to complex GitHub Actions configuration.
- MATLAB: Requires an environment variable
MATLAB_PATHto be set for initialization tests.
Sources: test/solidlsp/zig/test_zig_basic.py:126-140, test/solidlsp/swift/test_swift_basic.py:19-25, test/solidlsp/matlab/test_matlab_basic.py:27-35
Conclusion
The testing strategy is built around consistency and robustness. By abstracting language-specific behaviors into common integration tests, the project ensures that its semantic tools provide a unified interface regardless of the underlying programming language or LSP implementation. This is critical for the SerenaAgent to operate reliably as an "IDE for AI agents."