Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The testing strategy for OWASP Nettacker is designed to ensure the reliability, security, and accuracy of its automated penetration testing and information-gathering modules. The project employs a multi-layered approach involving automated unit testing, integration testing, and specialized validation of security modules defined in YAML.
Testing is integrated into the development lifecycle via CI/CD pipelines, utilizing standardized Python testing frameworks to maintain high code quality across the CLI, REST API, and modular scanning components. This strategy ensures that new features or bug fixes do not introduce regressions while verifying that the scanning engine accurately detects vulnerabilities and network configurations.
Sources: AGENTS.md:12-16, README.md:16-25, pyproject.toml:79-84
Core Testing Framework and Infrastructure
OWASP Nettacker uses pytest as its primary testing framework. The testing infrastructure is configured to support asynchronous operations and parallel execution to handle the multithreaded nature of the scanning engine.
Testing Tools and Configuration
The project leverages several pytest plugins and development tools to manage the test lifecycle:
- pytest-asyncio: Facilitates testing of asynchronous components, such as
aiohttpandasynciointegrations. - pytest-xdist: Enables parallel test execution using multiple CPU cores (
--numprocesses auto) to reduce total test time. - pytest-cov: Monitors and enforces code coverage through
coverage.py, ensuring that all parts of thenettackerpackage are exercised. - Ruff: Used for linting and formatting to ensure code style consistency before tests are executed.
Sources: AGENTS.md:12-16, pyproject.toml:58-62, pyproject.toml:79-84
Test Organization
The project follows a mirrored directory structure for its test suite, ensuring clear mapping between source code and test coverage.
| Test Location | Target Component |
|---|---|
tests/core/ | Engine logic and core libraries |
tests/lib/ | Shared utilities and helper functions |
tests/api/ | REST API endpoints and web server logic |
tests/test_yaml_regexes.py | Validation of module-specific regex patterns |
Sources: AGENTS.md:3-7, pyproject.toml:86
Automated Test Workflow
The testing workflow is automated through a series of commands and CI/CD integrations. Developers are expected to run pre-commit hooks and local tests before pushing code.
The diagram above illustrates the local development and validation flow required for contributors.
Sources: AGENTS.md:12-25, pyproject.toml:80-84
Specialized Module Validation
A critical component of Nettacker is its library of scanning and vulnerability modules, many of which are defined in YAML. The testing strategy includes specific validation for the Regular Expressions (Regex) used within these modules to detect vulnerabilities or services.
YAML Regex Testing Logic
The file tests/test_yaml_regexes.py contains automated tests that load every YAML file in nettacker/modules/vuln and nettacker/modules/scan. It extracts regex patterns from the "steps" and "response" sections and validates them against a dummy string to ensure they are syntactically correct and compilable.
The sequence diagram shows the process of validating scanning modules to prevent runtime errors during actual penetration testing.
Sources: tests/test_yaml_regexes.py:11-30, tests/test_yaml_regexes.py:80-90
Security and Vulnerability Handling
The testing strategy is complemented by a security policy focused on responsible disclosure and version maintenance.
- Triage: Security reports are triaged by project leaders to assess severity.
- Fix Development: Vulnerabilities found in the framework are fixed and tested internally before a patched release is issued.
- Regression Monitoring: New tests are required for every bug fix to prevent future regressions.
Sources: SECURITY.md:27-32, AGENTS.md:14-15
Summary of Testing Requirements
The following table summarizes the requirements for a successful test run and code contribution.
| Requirement | Command / Tool | Source of Truth |
|---|---|---|
| Linting | make pre-commit | pyproject.toml |
| Execution | poetry run pytest | tests/ directory |
| Coverage | --cov=nettacker | tool.pytest.ini_options |
| Python Support | 3.10 – 3.12 | pyproject.toml |
| YAML Validity | test_yaml_regexes.py | nettacker/modules/ |
Sources: AGENTS.md:12-25, pyproject.toml:42, pyproject.toml:79-84
Conclusion
The OWASP Nettacker testing strategy ensures that the automated penetration testing framework remains a stable and effective tool for security professionals. By combining standard Python unit testing with specialized YAML module validation and strict CI/CD guidelines, the project maintains high standards of technical accuracy and prevents the introduction of faulty vulnerability detection patterns.
Sources: AGENTS.md:38-42, README.md:16-25