Regex Parsing & Log Formats

Relevant source files

The following files were used as context for generating this wiki page:

• src/file_reader.rs
• README.md
• src/main.rs
• src/file_settings.rs
• src/memory_db.rs
• src/tabs/security.rs

LogUtil employs a flexible regex-based parsing system to transform raw log lines into structured data for real-time analysis. The system is designed to handle standard Nginx and Apache formats by default while allowing users to supply custom regular expressions and date formats through command-line arguments or an interactive configuration interface.

The core of this system resides in the FileReader component, which utilizes the regex-lite crate for high-performance pattern matching. Parsed data is mapped into a structured LogRecord and stored in an in-memory database for immediate visualization across various analytical tabs.

Sources: src/file_reader.rs:1-40, README.md:12-25

Core Parsing Architecture

The parsing logic is encapsulated within the FileReader struct. It manages the file path, the regex pattern, and the date format used to interpret timestamps. The process follows a strict flow from raw file input to structured database insertion.

Data Flow Overview

The following diagram illustrates how a raw log line is processed:

The system supports parallel processing of log lines in chunks of 1000 using the rayon crate to maintain high throughput even with large files.

Sources: src/file_reader.rs:43-150, src/main.rs:270-300

Supported Log Formats

LogUtil provides pre-configured regex patterns for the most common web server logs. These patterns utilize capture groups to extract specific fields such as IP address, timestamp, HTTP method, URL, status code, and response size.

Format	Regex Pattern	Default Date Format
Nginx (Default)	^(\S+) - ".+" \[(.*?)\] \d+\.\d+ "(\S+)" "(\S+) (\S+?)(?:\?.*?)? "	%d/%b/%Y:%H:%M:%S %z
Apache	^(\S+) \S+ \S+ \[.*?\] "\S+ (\S+?)(?:\?.*?)? \S+" \d+ \d+	%d/%b/%Y:%H:%M:%S %z
Custom	User-defined via --regex	User-defined via --date-format

Sources: README.md:325-345, src/main.rs:60-75

The Parsing Logic

The parse_line function in src/file_reader.rs is responsible for applying the regex and mapping capture groups to the LogEntryParams struct.

Field Extraction Mapping

The default regex extracts data into the following internal structure:

pub struct LogEntryParams {
    pub ip: String,
    pub url: String,
    pub log_line: String,
    pub timestamp: i64,
    pub request_type: String,
    pub request_domain: String,
    pub status_code: Option<u16>,
    pub response_size: Option<u64>,
    pub response_time: Option<f32>,
    pub user_agent: Option<String>,
}

Sources: src/file_reader.rs:12-30

Processing Steps

1. Regex Capture: The line is matched against the pattern. If it fails, the line is skipped and an error may be logged in debug mode.
2. Timestamp Parsing: Capture group 2 is converted from a string (e.g., 15/Oct/2023:14:30:01 +0000) to a Unix timestamp using chrono.
3. Request Line Decomposition: The request line (often combining method and path) is split to isolate the HTTP verb (GET, POST) and the URL path.
4. Domain Extraction: If the URL contains a protocol (http://), the domain is extracted; otherwise, it defaults to the first segment of the path or "unknown".

Sources: src/file_reader.rs:188-268

Security Pattern Identification

Beyond standard parsing, the system performs secondary analysis on the log_line to identify security threats. This does not change the structured format but adds metadata markers for the Security Tab.

The Security module looks for specific substrings within the raw log line, such as:

• SQL Injection: ', union, select, drop, exec.
• XSS: <script>, javascript:, onload=.
• Path Traversal: ../, /etc/passwd.

Sources: src/tabs/security.rs:240-300, src/memory_db.rs:435-460

Configuration via CLI and UI

Users can override the default parsing behavior using the following parameters:

CLI Option	Description	FileSettings Equivalent
--regex	Sets the pattern for line matching.	Regex Pattern
--date-format	Sets the chrono format string for timestamps.	Date Format
--count	Determines how many lines to read (0 = tail, -1 = all).	Count

Sources: src/main.rs:55-80, src/file_settings.rs:90-130

Conclusion

LogUtil's regex parsing system provides the foundation for its real-time analytics. By converting unstructured log files into a strongly-typed LogRecord format, the application enables high-speed indexing in the MemoryDB and rich visual feedback through the TUI. The modularity of the FileReader ensures that as new log formats emerge, they can be supported through simple regex configuration updates.