Performance Tuning & Considerations

Relevant source files

The following files were used as context for generating this wiki page:

• src/memory_db.rs
• tests/performance_tests.rs
• README.md
• src/tabs/performance.rs
• src/main.rs
• src/file_settings.rs
• Cargo.toml

Introduction

Performance Tuning & Considerations in LogUtil centers on the efficient processing and storage of large-scale Nginx access logs within an interactive Terminal User Interface (TUI). The system is designed to handle over 10,000 log entries in real-time while maintaining minimal CPU and memory footprints. This is achieved through a combination of optimized Rust data structures, efficient regex parsing, and proactive memory management policies.

The core architecture utilizes an in-memory database (MemoryDB) backed by thread-safe concurrent maps and atomic operations. This allows the tool to provide live updates, real-time analytics, and security threat detection without degrading the user experience or the performance of the host system.

Sources: README.md:52-57, src/memory_db.rs:25-50

Memory Management Strategy

LogUtil employs a multi-tiered strategy to manage memory usage, ensuring stability even when processing millions of log records.

Automatic Cleanup and Eviction

The system implements a FIFO (First-In-First-Out) eviction policy. By default, the MemoryDB is configured with a maximum record limit (defaulting to 10,000,000 in code, though documented as 10,000 for standard cleanup triggers). When this limit is reached, the evict_old_records function removes half of the oldest records to reclaim space.

Sources: src/memory_db.rs:80-112, README.md:214-218

Optimized Data Structures

To minimize memory overhead per entry, LogUtil uses specific Rust types:

• f32 instead of f64: Used for response_time to reduce the size of each LogRecord.
• DashMap: A high-performance concurrent hash map used for the primary record storage and indexing, reducing locking contention.
• ID-based Indexing: Indices for IP addresses and URLs store only the u64 record ID rather than duplicating the full record data.

Sources: src/memory_db.rs:18-35, Cargo.toml:44

Data Retention Policy

Entries older than 20 minutes are automatically removed when the total count exceeds 10,000 to maintain a rolling window of recent activity. Users can disable this behavior using the --no-clear flag for long-term analysis.

Sources: README.md:214-218, src/file_settings.rs:88-93

Processing Speed and Throughput

The application is optimized for high-speed ingestion and querying of log data.

Concurrent Ingestion Flow

The following diagram illustrates how log data moves from the file system into the optimized in-memory storage.

The system uses AtomicUsize for ID generation and concurrent maps to ensure that ingestion does not block the TUI rendering loop.

Sources: src/memory_db.rs:71-78, src/main.rs:281-304

Query Optimization

Indices are maintained for common query paths:

Index Type	Data Structure	Purpose
ip_index	DashMap<String, Vec<u64>>	Fast lookup by IP for the Detailed tab.
url_index	DashMap<String, Vec<u64>>	Fast lookup by URL for Overview.
status_code_index	DashMap<u16, Vec<u64>>	Used for Error Tab and Security analysis.
error_records_cache	DashMap<u64, bool>	Dedicated cache for 4xx/5xx status codes.

Sources: src/memory_db.rs:36-50

Performance Metrics and Monitoring

LogUtil provides a dedicated Performance Tab to monitor the health of the Nginx server and the tool itself.

Key Performance Indicators (KPIs)

• Requests Per Second (RPS): Calculated based on total requests over a rolling 60-second window.
• Response Time Statistics: Tracks average, minimum, and maximum response times across sampled records.
• Slow Request Identification: Automatically flags requests with response times exceeding 1.0 seconds.

Sources: src/tabs/performance.rs:35-40, src/memory_db.rs:242-255

Performance Metrics Architecture

Build-Time Optimizations

The Cargo.toml configuration includes specific profiles for release builds to maximize runtime performance and minimize binary size.

[profile.release]
opt-level = "z"        # Optimize for size
lto = true             # Link Time Optimization
codegen-units = 1      # Better optimization across units
panic = "abort"        # Reduce binary size
strip = true           # Remove debug symbols

Sources: Cargo.toml:48-53

Benchmarking Results

Performance tests conducted on the MemoryDB implementation demonstrate the following capabilities:

• Large Datasets: Successful handling of 1,000,000+ records in unit tests.
• Search Latency: IP and URL lookups remain performant even with millions of records due to Vec<u64> indexing.
• Cache Acceleration: Repeated calls to get_top_ips and get_top_urls show significant speedups when utilizing internal caching mechanisms.

Sources: tests/performance_tests.rs:6-60, tests/performance_tests.rs:188-210

Summary

Performance in LogUtil is a balance between real-time visibility and resource conservation. By utilizing thread-safe concurrent data structures, efficient FIFO eviction, and specialized status code caching, the application provides deep insights into Nginx logs without becoming a bottleneck on the monitoring system. Developers should prioritize maintaining the efficiency of the MemoryDB indices when adding new analytical features.