Memory Database

Relevant source files

The following files were used as context for generating this wiki page:

• src/memory_db.rs
• src/tabs/mod.rs
• src/app.rs
• tests/performance_tests.rs
• tests/memory_db_tests.rs
• README.md

The Memory Database is a core component of the LogUtil project, designed as a high-performance, thread-safe, in-memory storage system for indexed log data. It provides the backend infrastructure required to process thousands of log entries in real-time, offering optimized search capabilities and statistical aggregation for various TUI tabs such as Overview, Security, and Performance.

Sources: src/memory_db.rs:33-52, README.md:38-42

Core Architecture

The database utilizes DashMap for concurrent, lock-free access to data and specialized indices. It is designed to minimize memory overhead by using smaller data types (e.g., f32 for response times) and storing only unique IDs in its secondary indices.

Data Structures

The primary unit of storage is the LogRecord, which encapsulates parsed log information.

Field	Type	Description
id	u64	Unique identifier for the record.
ip	String	Source IP address.
url	String	Requested URL path.
timestamp	i64	Unix timestamp of the request.
status_code	Option<u16>	HTTP response status code.
response_size	Option<u64>	Size of the response in bytes.
response_time	Option<f32>	Time taken to process the request (optimized to f32).
log_line	String	The original raw log string.

Sources: src/memory_db.rs:7-24

Database Components Diagram

The following diagram illustrates the relationship between the main storage, indexing structures, and statistics tracking within MemoryDB.

Data Management Logic

Insertion and Indexing Flow

When a new log entry is parsed, it is inserted into the global database instance. The system automatically increments the internal ID, updates multiple indices for O(1) or O(log N) lookups, and refreshes global statistics.

Memory Management and Eviction

To prevent unbounded memory consumption, the database implements a FIFO (First-In-First-Out) eviction policy. When the record count reaches the max_records threshold (defaulting to 10 million), the system removes approximately half of the oldest records and cleans up their corresponding entries in all indices.

Sources: src/memory_db.rs:110-128, tests/performance_tests.rs:114-118

Search and Analytics Features

Specialized Query Methods

The database provides optimized methods for retrieving the most relevant data for the TUI:

• Top Statistics: get_top_ips and get_top_urls allow for sorted retrieval of high-frequency items with a configurable limit.
• Security Analysis: Methods like get_suspicious_ips and get_attack_patterns scan specifically for known malicious strings (e.g., "union select", "etc/passwd") within log lines associated with error status codes.
• Performance Monitoring: get_slow_requests_with_limit identifies requests exceeding a specific duration threshold (e.g., > 1.0s).

Sources: src/memory_db.rs:194-220, src/memory_db.rs:347-380, src/memory_db.rs:311-331

Global Access Pattern

The application typically interacts with the database through a lazy-initialized global singleton.

/// Global singleton instance for database access
pub static GLOBAL_DB: std::sync::LazyLock<Arc<MemoryDB>> = std::sync::LazyLock::new(|| Arc::new(MemoryDB::new())); 

Sources: src/memory_db.rs:460-461, src/app.rs:247-251

Performance Characteristics

Based on performance testing, the MemoryDB is optimized for high throughput.

Operation	Performance/Note
Insertion	Handles ~1,000,000 records with sub-second IP search.
Memory Usage	Approximately 500-600 MB for 1 million records including indices.
Search (IP/URL)	Result sets are truncated (usually to 1000) to ensure UI responsiveness.
Concurrency	Uses DashMap to avoid global lock contention across threads.

Sources: tests/performance_tests.rs:10-50, tests/performance_tests.rs:90-105

Summary

The MemoryDB provides a high-performance foundation for LogUtil's real-time analysis. By combining DashMap for concurrency, specific indices for rapid UI updates, and a proactive eviction policy, it ensures the application remains responsive even when monitoring high-traffic Nginx access logs. Its centralized nature allows different UI modules—from security auditing to performance tracking—to share a consistent and efficient view of the system state.