Bots Tab

Relevant source files

The following files were used as context for generating this wiki page:

• src/tabs/bots.rs
• src/tabs/base.rs
• src/memory_db.rs
• src/app.rs
• src/main.rs
• README.md

The Bots Tab is a specialized analysis module within the LogUtil terminal interface designed to identify, classify, and monitor automated traffic. It provides real-time visibility into bot activity by scanning Nginx access logs for specific User-Agent strings associated with search engine crawlers, web spiders, and scrapers.

Sources: README.md:58, README.md:204-208

Overview and Purpose

The primary purpose of the Bots Tab is to distinguish between human and automated traffic. It enables administrators to quantify bot behavior and behavior patterns, helping to identify both legitimate search engine activity (e.g., Googlebot, Bingbot) and potentially malicious scrapers. The tab is optional and can be enabled via the --enable-bots command-line flag or through the interactive settings menu.

Sources: README.md:204-208, src/main.rs:113, src/file_settings.rs:141-145

Architecture and Data Flow

The Bots Tab follows the standard tab architecture of the LogUtil application, implementing the Tab trait. It relies on the GLOBAL_DB (a MemoryDB instance) to retrieve aggregated statistics and User-Agent data.

Data Acquisition Flow

The flow of data from the raw log file to the Bots Tab display is as follows:

The MemoryDB performs a case-insensitive search within the user_agent field of LogRecord objects to identify bots. It looks for keywords such as "bot", "crawler", or "spider".

Sources: src/memory_db.rs:271-285, src/tabs/bots.rs:34-40

Components and Logic

UI Structure

The BotsTab UI is divided into two main vertical sections using ratatui layout constraints:

1. Summary Panel: A top bar displaying high-level metrics.
2. Top Bot Types Table: A detailed list of User-Agents and their classifications.

Sources: src/tabs/bots.rs:32-37

Summary Metrics

The summary panel utilizes the get_bot_stats() method from the database to display:

Metric	Description
Bot IPs	The count of unique IP addresses identified as bots.
Bot Types	The count of distinct bot User-Agent strings detected.
Bot URLs	The count of unique URLs accessed by bot traffic.

Sources: src/tabs/bots.rs:35-36, src/memory_db.rs:271-285

Bot Classification Logic

The system classifies User-Agents into four primary categories based on string matching:

Implementation Details

Key Methods and Functions

Function	File	Description
get_bot_stats()	src/memory_db.rs	Iterates through records to count unique IPs and URLs containing bot-related keywords in the User-Agent.
get_top_user_agents(limit)	src/memory_db.rs	Aggregates and sorts User-Agents by frequency of occurrence.
draw_bots_tab()	src/tabs/bots.rs	Handles the rendering logic for the summary paragraph and the bot activity table.
handle_input()	src/tabs/bots.rs	Manages vertical navigation (Up/Down arrows) within the User-Agent table.

Sources: src/memory_db.rs:271-300, src/tabs/bots.rs:32-105

Data Structure: LogRecord

The bot detection relies on the user_agent field within the LogRecord structure.

pub struct LogRecord {
    pub id: u64,
    pub ip: String,
    pub url: String,
    pub timestamp: i64,
    // ... other fields
    pub user_agent: Option<String>,
    pub log_line: String,
}

Sources: src/memory_db.rs:8-23

Summary

The Bots Tab provides a critical layer of observability for Nginx logs by identifying automated agents. By combining real-time keyword-based classification with aggregated frequency statistics from the MemoryDB, it allows users to quickly distinguish between legitimate search engine indexing and potentially aggressive scraping activity directly from the TUI.

Sources: README.md:204-208, src/tabs/bots.rs:1-30