Querying and Filtering

Querying and filtering in Cyberpath Sentinel are designed around its core principle: "the filesystem is the database." The initial approach prioritizes simplicity, transparency, and auditability over raw performance, leveraging standard filesystem operations. As the system evolves, more advanced features like in-memory caching, lazy indexing, and full-text search are planned to improve performance for larger datasets without sacrificing the foundational design goals.

The system is not intended for high-frequency trading or real-time analytics but is optimized for use cases where data transparency is paramount, such as audit logs, compliance rule management, and certificate storage. The query mechanism is planned to evolve through several phases, starting with basic in-memory filtering and progressing to a sophisticated engine with multiple indexing strategies.

Sources: README.md:10-12, README.md:17-25, IMPLEMENTATION_PLAN.md:14-20

Core Query Architecture

The query architecture is composed of a QueryEngine that interacts with the CollectionManager and a planned IndexManager to retrieve documents. Queries are constructed using a QueryBuilder pattern, which allows for a flexible and composable interface.

Sources: IMPLEMENTATION_PLAN.md:23-37, IMPLEMENTATION_PLAN.md:220-223

Query Flow

The following diagram illustrates the planned flow of a query from the client to the filesystem.

This diagram shows that the QueryEngine first consults the IndexManager. If a suitable index exists, it's used to find document IDs. Otherwise, the CollectionManager falls back to scanning files directly. The Caching Layer is checked to avoid redundant file I/O.

Sources: IMPLEMENTATION_PLAN.md:23-37, IMPLEMENTATION_PLAN.md:112-115, IMPLEMENTATION_PLAN.md:160-163

Basic Filtering

The most fundamental query operation is filtering, which involves iterating through all documents in a collection and applying a predicate function. In its initial implementation, this is an O(n) operation where n is the number of documents, as it requires scanning all files in the collection's directory.

A basic filter operation is exposed via the filter method on a Collection.

// Query with filters
let admins = users.filter(|doc| {
    doc["role"].as_str().unwrap_or("") == "admin"
}).expect("Failed to filter");

Sources: README.md:92-96, README.md:148

Planned Query Structures

The IMPLEMENTATION_PLAN.md outlines a set of data structures to enable more complex and structured queries beyond simple closure-based filtering.

Query and Filter Definitions

The Query struct encapsulates all parts of a request, including filters, sorting, and pagination. The Filter enum defines a recursive structure for building complex logical conditions.

pub struct Query {
    pub filters: Vec<Filter>,
    pub sort: Option<SortOrder>,
    pub limit: Option<usize>,
    pub offset: Option<usize>,
}

pub enum Filter {
    Equals(String, Value),
    GreaterThan(String, Value),
    LessThan(String, Value),
    Contains(String, String),
    In(String, Vec<Value>),
    And(Box<Filter>, Box<Filter>),
    Or(Box<Filter>, Box<Filter>),
}

Sources: IMPLEMENTATION_PLAN.md:126-143

Query Builder

A QueryBuilder is proposed to provide a fluent API for constructing queries.

pub struct QueryBuilder {
    filters: Vec<Filter>,
    sort: Option<(String, SortOrder)>,
    limit: Option<usize>,
    offset: Option<usize>,
    projection: Option<Vec<String>>,
}

impl QueryBuilder {
    pub fn filter(mut self, field: &str, op: Operator, value: Value) -> Self
    pub fn sort(mut self, field: &str, order: SortOrder) -> Self
    pub fn limit(mut self, limit: usize) -> Self
    pub fn offset(mut self, offset: usize) -> Self
    pub fn projection(mut self, fields: Vec<&str>) -> Self
    pub async fn execute(self) -> io::Result<QueryResult>
}

Sources: IMPLEMENTATION_PLAN.md:225-244

The following class diagram shows the relationships between these planned query components.

Performance and Optimization

A key challenge for Sentinel is query performance on large collections. The project plans to address this through several optimization strategies, primarily indexing and caching.

Sources: README.md:52-54, IMPLEMENTATION_PLAN.md:368-371

Indexing Engine

To avoid full collection scans, a lazy-built indexing engine is a core part of the planned implementation. An index will be created on a field the first time it is queried.

Indexing Components

The IndexManager is responsible for creating, managing, and querying indices. Different index types are planned to support various query patterns.

Sources: IMPLEMENTATION_PLAN.md:53-57, IMPLEMENTATION_PLAN.md:160-175

The following diagram illustrates the lazy indexing process.

This "quick win" strategy makes the first query on a field slow but subsequent queries fast.

Sources: IMPLEMENTATION_PLAN.md:374-381, IMPLEMENTATION_PLAN.md:386-396

Indexing Dependencies

The implementation plan suggests using external crates to build the indexing engine.

These dependencies are not yet present in the main Cargo.toml but are part of the plan.

Sources: IMPLEMENTATION_PLAN.md:188-190, Cargo.toml

Caching Layer

An in-memory LRU (Least Recently Used) cache is planned to store frequently accessed documents, reducing disk I/O. The cache will be managed by the Store and used by the QueryEngine.

Key characteristics of the planned cache:

• Eviction Policy: LRU
• Concurrency: Implemented with DashMap for high-performance concurrent access.
• Configuration: Cache size will be configurable.

The lru and arc-swap crates are noted as potential dependencies for this feature.

Sources: README.md:151, IMPLEMENTATION_PLAN.md:76-79, IMPLEMENTATION_PLAN.md:144-158

Summary

Sentinel's approach to querying and filtering is a direct reflection of its core design philosophy. It begins with a simple, transparent, but slow method of file scanning. The roadmap includes a clear, phased evolution towards a high-performance query system featuring lazy indexing, multiple index types, and an in-memory caching layer. This strategy allows the system to scale while retaining its fundamental promises of auditability and operational simplicity. The implementation is planned in phases, prioritizing durability and concurrency before focusing on performance optimizations.

Sources: README.md:109-113, IMPLEMENTATION_PLAN.md:292-332