Advanced Filtering System

Relevant source files

The following files were used as context for generating this wiki page:

• packages/core/src/types/filter.ts
• packages/core/src/managers/filter-manager.ts
• packages/core/src/utils/filter-serialization.ts
• plans/015-filternode-core-serialization.md
• plans/006-core-contract-v2-design.md
• plans/004-validate-url-state.md
• packages/core/src/utils/url-serialization.ts
• plans/016-filternode-state-layer.md

The Advanced Filtering System in Better Tables provides a type-safe, multi-layered architecture for managing complex data queries. It supports a wide range of operators across diverse data types, including text, numbers, dates, and options, while enabling sophisticated boolean logic through recursive filter groups. The system is designed to handle state management, validation, and URL-based persistence seamlessly.

Central to this system is the FilterNode structure, which allows for both flat filter arrays and nested AND/OR logic. This architecture ensures that filtering criteria can be easily authored in the UI, synchronized with the browser's URL state, and translated into efficient database queries via adapters.

Sources: plans/015-filternode-core-serialization.md, plans/006-core-contract-v2-design.md, README.md:144-165

Core Architecture and Data Structures

The system is built around a recursive tree structure called a FilterNode. A node can either be a simple FilterState (a leaf node targeting a specific column) or a FilterGroupNode (a branch node containing other nodes).

Filter Nodes and Groups

• FilterState: The basic unit of filtering. It contains a columnId, a type (e.g., text, number), an operator (e.g., contains, equals), and values.
• FilterGroupNode: A container that applies logical operators (and or or) to its children. To prevent excessive complexity and performance degradation, nesting is typically capped at a depth of 3.

Sources: packages/core/src/types/filter.ts:231-262, plans/015-filternode-core-serialization.md:§1.1-1.2

Component Relationship Diagram

This diagram illustrates how filters are structured within the core logic of the system.

Filter Types and Operators

Better Tables defines a strict set of operators categorized by the data type of the column. This ensures type safety when authoring filters and constructing database predicates.

Filter Category	Supported Types	Common Operators
Text	text, email, url, phone	contains, equals, startsWith, endsWith, isEmpty, isNotEmpty
Numeric	number, currency, percentage	equals, notEquals, greaterThan, lessThan, between
Temporal	date	is, isNot, before, after, between
Categorical	option, multiOption	is, isNot, isAnyOf, isNoneOf, includes, excludes
Logical	boolean	isTrue, isFalse

Sources: packages/core/src/types/filter.ts:10-57, README.md:151-163, plans/031-filter-authoring-typesafety.md:§1

State Management and Lifecycle

The FilterManager is responsible for the local lifecycle of filters within a table instance. It handles the addition, removal, and clearing of filters while ensuring all changes are validated against the column definitions.

Filtering Data Flow

The following sequence diagram represents how a filter change initiated by a user propagates through the system.

Serialization and URL Persistence

To enable shareable views, the filtering system synchronizes its state with the browser URL. This involves a custom serialization process that utilizes compression to keep URLs concise.

Wire Formats

• Legacy Format (c:): A flat, implicit-AND FilterState[] array.
• Modern Format (c2:): A versioned format capable of representing recursive FilterNode trees.

Validation Semantics (Fail-Closed)

When deserializing from the URL, the system applies a "fail-closed" policy to prevent crashes from malformed or malicious input:

1. Normalization: Nodes with unknown logic or invalid leaves are dropped.
2. Unwrapping: Groups with a single child are unwrapped into that child node.
3. Depth Capping: Subtrees exceeding the depth limit (default 3) are dropped.
4. Value Protection: User data within meta or values keys is protected from key-renaming mangling during decompression (Fix for CORE-06).

Sources: packages/core/src/utils/filter-serialization.ts:63-90, plans/015-filternode-core-serialization.md:§1.3-1.4, plans/004-validate-url-state.md:Step 2

Integration with Database Adapters

The filtering system acts as an intermediate layer between the UI and the database. Adapters (such as Drizzle) translate the FilterNode tree into SQL predicates.

• Recursive Translation: Adapters walk the filter tree, converting logical groups into SQL AND/OR clauses and leaf filters into WHERE conditions.
• Capability Discovery: Adapters advertise their filtering capabilities via AdapterMeta, such as supportsFilterGroups and maxGroupDepth. If an adapter does not support groups, the core system may flatten the tree or reject the query to ensure semantic accuracy.

Sources: plans/006-core-contract-v2-design.md:Step 1.5, plans/015-filternode-core-serialization.md:§1.2

The Advanced Filtering System provides a robust foundation for data interaction in Better Tables, combining a flexible recursive logic model with strict type safety and optimized persistence for modern web applications.