Database Management

Relevant source files

The following files were used as context for generating this wiki page:

• nettacker/database/db.py
• nettacker/database/models.py
• nettacker/database/mysql.py
• nettacker/config.py
• nettacker/database/readme.md
• tests/database/test_db.py
• nettacker/core/lib/base.py

Database Management in OWASP-Nettacker is a multi-engine system designed to store scan results, logs, and report metadata. It supports both relational databases (MySQL, PostgreSQL) and lightweight local storage (SQLite). The system is architected to handle high-concurrency write operations during multi-threaded scans while maintaining data integrity through transaction management and retry logic.

The core purpose of the database module is to facilitate drift detection by comparing past scan results with current ones, enabling users to identify new hosts, open ports, or newly discovered vulnerabilities over time. It provides the backend for both the CLI and the Web UI/API components. Sources: README.md, nettacker/database/readme.md

Core Architecture and Engine Support

Nettacker utilizes a dual-layer approach for database interactions. It employs SQLAlchemy as an Object-Relational Mapper (ORM) for MySQL and PostgreSQL, and traditionally used APSW (Another Python SQLite Wrapper) for optimized SQLite operations, although this is configurable. Sources: nettacker/config.py:118-135, nettacker/config.py:171

Supported Database Engines

Engine	Driver/Library	Configuration Context
SQLite	APSW or SQLAlchemy	Default local storage; uses WAL (Write-Ahead Logging) mode.
MySQL	PyMySQL	Remote/Production storage; requires username, password, host, and port.
PostgreSQL	Psycopg2	Remote/Production storage; supports SSL modes for encrypted connections.

Sources: nettacker/config.py:118-135, nettacker/database/mysql.py:12-21

Connection Management Flow

The following diagram illustrates how the system initializes a connection based on the configured engine:

Data Models and Schema

The database schema is defined in models.py and consists of three primary tables: reports, scan_events, and temp_events. Sources: nettacker/database/readme.md

Primary Entities

Table	Purpose	Key Fields
reports	Stores metadata for completed scan jobs.	id, date, scan_unique_id, report_path_filename, options
scan_events	Stores persistent results from modules.	id, target, date, module_name, scan_unique_id, port, event, json_event
temp_events	Stores intermediate data used for module dependencies.	id, target, date, module_name, scan_unique_id, event_name, data

Sources: tests/database/test_db.py:182-195, tests/database/test_db.py:326-338, tests/database/test_db.py:441-450

Transaction Logic and Data Flow

Logging results into the database involves a structured process to ensure data is correctly formatted and sensitive information is scrubbed before persistence. Sources: nettacker/core/lib/base.py:108-144

Result Submission Process

When a scan module identifies a vulnerability or open port, the BaseEngine processes the event:

1. Sensitive Data Scrubbing: HTTP headers like authorization, cookie, and x-api-key are removed.
2. Formatting: The event is converted to a JSON/YAML string.
3. Submission: Data is sent to submit_logs_to_db.
4. Transaction Handling:
   • For SQLite/APSW: The system manually issues BEGIN and COMMIT commands.
   • For SQLAlchemy: The system uses session commits.
5. Retry Mechanism: If the database is locked (common in SQLite during heavy threading), the system retries up to max_submit_query_retry times (default 100).

Temporary Events and Module Dependencies

Some modules depend on the output of previous modules (e.g., a vulnerability scan needing results from a port scan). The temp_events table handles this internal data flow. Sources: nettacker/core/lib/base.py:48-57

Dependency Resolution

1. Storage: A module saves intermediate data using submit_temp_logs_to_db with a specific event_name.
2. Retrieval: A dependent module calls get_dependent_results_from_database.
3. Polling: The system polls the temp_events table until the required data appears or a timeout occurs.

Sources: nettacker/core/lib/base.py:51-57, nettacker/core/lib/base.py:202-205

Search and Reporting Interface

The database management system provides several utility functions for retrieving data for report generation and the Web UI. Sources: tests/database/test_db.py:615-680

• search_logs: Supports paginated searching of scan targets. It uses LIKE queries (e.g., %192.168%) and returns grouped results including module information and discovered ports.
• logs_to_report_html: Queries the database to build a comprehensive HTML report, including data for D3.js graphs.
• last_host_logs: Fetches the most recent activity for all hosts, used in the "Crawler" or "Dashboard" views of the Web UI.
• remove_old_logs: A maintenance function that deletes previous records for a target to prevent redundant data, while preserving logs needed for the current comparison/drift detection.

Sources: tests/database/test_db.py:218-232, tests/database/test_db.py:535-548, tests/database/test_db.py:648-660

Summary

OWASP-Nettacker's database management system acts as the central nervous system for scan data. By abstracting the storage layer through a combination of SQLAlchemy and APSW, it balances the need for high-performance local scanning with the scalability required for enterprise-grade deployments. Its robust transaction retry logic and sensitive data filtering ensure that scan results are preserved reliably and securely.