Workspace Scanner & Filtering
Relevant source files
The following files were used as context for generating this wiki page:
The Workspace Scanner & Filtering module is a core component of the chkpt system responsible for discovering files within a workspace while applying specific exclusion rules. Its primary purpose is to identify which files should be included in a checkpoint "save" operation or evaluated during a "restore" comparison. The scanner supports both sequential and parallel directory walking, ensuring high performance even in large repositories.
The scanner differentiates between regular source files and dependency directories (like node_modules or .venv), allowing users to optionally include or exclude heavy build artifacts and dependencies. It uses standard filesystem metadata to track file sizes, modification times, and device/inode information, which is critical for the system's incremental save and hardlink detection logic.
Sources: crates/chkpt-core/src/scanner/walker.rs:9-41, README.md:15-25
Architecture and Components
The scanning system is built upon a layered architecture that includes a high-level orchestration layer, a walking engine, and a filtering matcher.
Data Structures
The primary data structure used across the scanner is the ScannedFile (or JsScannedFile in NAPI bindings). It encapsulates all necessary filesystem metadata.
| Field | Type | Description |
|---|---|---|
relative_path | String | Path relative to the workspace root (forward-slash normalized). |
absolute_path | PathBuf | Full path on the host filesystem. |
size | u64 | File size in bytes. |
mtime_secs | i64 | Last modification time (seconds). |
mtime_nanos | i64 | Last modification time (nanoseconds). |
device | Option<u64> | Device ID (used for hardlink detection). |
inode | Option<u64> | Inode number (used for hardlink detection). |
mode | u32 | Unix file permissions and mode. |
is_symlink | bool | Whether the path is a symbolic link. |
Sources: crates/chkpt-core/src/scanner/walker.rs:180-234, crates/chkpt-napi/src/scanner.rs:5-14
Scanning Flow
The scanner can operate in two modes: Sequential (walk) and Parallel (walk_parallel). The parallel mode utilizes the ignore crate's parallel visitor pattern to distribute directory traversal across multiple CPU threads.
The sorting step is critical because it ensures that the resulting file list is deterministic, which is required for stable tree building during the save operation.
Sources: crates/chkpt-core/src/scanner/walker.rs:13-90, crates/chkpt-core/src/ops/save.rs:326-335
Filtering and Ignore Logic
The scanner applies multiple layers of filtering to determine if a file should be tracked.
Ignore Rules Priority
- Hardcoded Defaults: The scanner always excludes
.git/and the internal.chkpt/storage directory. - Dependency Filtering: Directories like
node_modules,.venv, andtargetare excluded by default unless theinclude_depsflag is explicitly set. - User Configuration: Rules defined in a
.chkptignorefile (using.gitignoresyntax) are applied.
Dependency Directories
The system recognizes several standard dependency and build artifact patterns for filtering:
- Node.js:
node_modules - Python:
.venv,venv,__pypackages__,.tox,.nox - Java/JVM:
.gradle,.m2 - Rust:
target(in certain contexts)
Sources: crates/chkpt-core/src/ops/restore.rs:431-443, crates/chkpt-core/tests/scanner_test.rs:40-100
Implementation Details
Parallel Traversal
In parallel mode, the system uses a CollectBuilder to spawn CollectVisitor instances. Each visitor maintains a local_files buffer to minimize lock contention on the primary results vector.
Path Normalization
The scanner normalizes paths to use forward slashes (/) even on Windows. This ensures that checkpoints created on one operating system can be restored or compared on another without path mismatch errors.
fn relative_path(root: &Path, path: &Path) -> String {
let relative = path.strip_prefix(root).unwrap_or(path);
if cfg!(windows) {
relative.to_string_lossy().replace('\\', "/")
} else {
relative.to_string_lossy().into_owned()
}
}
Sources: crates/chkpt-core/src/scanner/walker.rs:105-112
Hardlink and Symlink Handling
- Symlinks: The scanner uses
symlink_metadatato avoid following symbolic links to external locations. It records the link itself and its target content hash. - Hardlinks: On Unix systems, the scanner captures the
deviceandinode. During a save operation, this allows the system to detect that two paths point to the same data on disk, enabling content deduplication even before hashing.
Sources: crates/chkpt-core/src/scanner/walker.rs:180-205, crates/chkpt-core/src/ops/save.rs:252-277
Integration in Operations
The scanner is utilized by two major operations:
- Save Operation: Scans the workspace to find files to hash and compress. It uses the
include_depsflag fromSaveOptions. - Restore Operation: Scans the current workspace to compare it against a target snapshot. It intelligently detects if the target snapshot includes dependencies; if it does, the scanner automatically includes dependency directories during the "current state" scan to ensure an accurate diff.
Conclusion
The Workspace Scanner is the "eyes" of the chkpt system. By combining high-performance parallel walking with flexible filtering rules and robust metadata collection, it provides the foundation for fast, incremental, and content-addressable snapshots of complex development environments.