File Watcher & Incremental Indexer

Relevant source files

The following files were used as context for generating this wiki page:

• crates/arbor-watcher/src/lib.rs
• crates/arbor-watcher/src/indexer.rs
• crates/arbor-server/src/sync_server.rs
• crates/arbor-cli/src/commands.rs
• crates/arbor-watcher/README.md

The File Watcher and Incremental Indexer system provides real-time monitoring and semantic synchronization for the Arbor ecosystem. Its primary purpose is to maintain an up-to-date Code Property Graph by detecting filesystem modifications and performing targeted re-parsing of affected files. This ensures that architectural visualizations and AI-driven impact analyses remain accurate as the codebase evolves.

This system is composed of two main layers: a directory crawler that builds the initial graph using persistent caching, and a reactive watcher that handles live updates. By utilizing debounced event handling and .gitignore awareness, the indexer balances high responsiveness with computational efficiency, only updating "dirty" nodes in the graph when necessary.

Sources: crates/arbor-watcher/src/lib.rs:1-12, crates/arbor-watcher/README.md:16-24

System Architecture

The architecture follows a pipeline where filesystem events are filtered, debounced, and eventually processed by the core parsing engine to update the shared graph state.

The diagram illustrates the data flow from a raw filesystem event to a semantic graph update.
Sources: crates/arbor-watcher/README.md:26-30, crates/arbor-server/src/sync_server.rs:252-260

Initial Directory Indexing

Indexing is performed via index_directory, which walks a root path to build an initial ArborGraph. This process uses the ignore crate to respect .gitignore, global git ignores, and hidden file rules.

Caching and Incremental Logic

To optimize performance, the indexer supports a persistence layer through GraphStore. It compares the current Modification Time (mtime) of a file against a cached value to determine if re-parsing is required.

• Cache Hit: If mtimes match, nodes are loaded directly from the sled database.
• Cache Miss: If mtimes differ or the file is new, the file is sent to arbor_core::parse_file for a fresh extraction.
• Cleanup: Any files present in the cache but missing from the current filesystem walk are purged to handle deletions.

Sources: crates/arbor-watcher/src/indexer.rs:66-160

Indexing Results and Options

The indexing operation returns an IndexResult containing performance metrics and error logs.

Field	Type	Description
files_indexed	usize	Number of files freshly parsed.
cache_hits	usize	Number of files loaded from persistence.
nodes_extracted	usize	Total count of semantic nodes in the resulting graph.
duration_ms	u64	Total execution time in milliseconds.
errors	Vec<(String, String)>	List of file paths and their associated parse error messages.

Sources: crates/arbor-watcher/src/indexer.rs:16-33

Real-Time Synchronization Server

The SyncServer manages the lifecycle of live updates. It integrates a RecommendedWatcher from the notify crate to monitor the filesystem for Changed, Created, and Deleted events.

Debouncing Mechanism

To prevent thrashing during rapid file saves (e.g., during a "Save All" operation or git branch switch), the server implements a debouncing logic using a HashMap to track pending paths and timestamps.

1. Events are received and stored with a timestamp.
2. The server periodically checks the pending map.
3. Only events older than the debounce_ms threshold (default 150ms) are forwarded to the indexer.

Sources: crates/arbor-server/src/sync_server.rs:356-407

Background Indexer Loop

A dedicated task consumes debounced WatcherEvent instances. When a change is detected:

• The SyncServer removes old nodes associated with the file path from the SharedGraph.
• The file is re-parsed into new CodeNode and SymbolRelation objects.
• The new data is integrated into the graph, and a GraphUpdate message is broadcast via WebSockets to connected clients (like the Visualizer).

The sequence shows the reactive loop for live code updates.
Sources: crates/arbor-server/src/sync_server.rs:418-500

CLI Integration

The Arbor CLI provides several commands to interact with the watcher and indexer systems:

• arbor index: Performs a manual walk of the directory. It provides flags for --no-cache and --follow-symlinks.
• arbor watch: Enters a loop that sleeps for 2 seconds between manual index_directory calls to detect changes (standard watch mode).
• arbor bridge: Starts the SyncServer alongside an MCP server, enabling both live visualization and AI-driven analysis.

Sources: crates/arbor-cli/src/commands.rs:48-60, crates/arbor-cli/src/commands.rs:880-920

Configuration

The SyncServerConfig defines the behavior of the real-time system:

pub struct SyncServerConfig {
    pub addr: SocketAddr,        // Default: 127.0.0.1:8080
    pub watch_path: PathBuf,     // Root directory
    pub debounce_ms: u64,        // Default: 150ms
    pub extensions: Vec<String>, // Default: ts, tsx, js, jsx, rs, py
}

Sources: crates/arbor-server/src/sync_server.rs:48-68

Conclusion

The File Watcher & Incremental Indexer forms the reactive backbone of Arbor. By combining exhaustive directory crawling with event-driven updates, it maintains a semantically accurate representation of the codebase. The use of persistent caching in indexer.rs and debounced WebSocket broadcasting in sync_server.rs ensures the system remains performant even in large-scale projects with hundreds of thousands of nodes.