Data Flow & Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Data Flow & Pipeline in Better Tables is a type-safe, database-agnostic architecture designed to transform high-level declarative column definitions into optimized database queries and interactive UI components. The system operates on a "one-liner" principle: column definitions in @better-tables/core are processed by an adapter (e.g., @better-tables/adapters-drizzle) to generate DB queries, the results of which are then rendered and managed by @better-tables/ui with full URL state synchronization.
Sources: CLAUDE.md:27-31, apps/marketing/src/components/home/pipeline.tsx:68-71, README.md:38-40
Architecture Overview
The pipeline is structured as a monorepo with a clear separation of concerns, ensuring that logical components like type systems, state managers, and database adapters interact through well-defined contracts.
System Components
| Component | Responsibility |
|---|---|
Core (@better-tables/core) | Defines the schema, column builders, and the betterTables factory instance. Manages state for filters, sorting, and pagination. |
| Adapters | Translates high-level table requests into ORM or database-specific commands (e.g., SQL for Drizzle). |
UI (@better-tables/ui) | React-based layer that renders the table and provides interactive controls for state manipulation. |
CLI (@better-tables/cli) | Manages the distribution of UI components into consumer projects (shadcn-style). |
Sources: README.md:154-167, CLAUDE.md:8-25
Data Flow Diagram
The following diagram illustrates the lifecycle of a request from definition to rendering.
The flow ensures that state changes (like applying a filter) re-trigger the adapter's fetch logic to retrieve updated, typed data.
Sources: CLAUDE.md:27-31, apps/marketing/src/components/home/pipeline.tsx:68-71
The Transformation Pipeline
The core of the pipeline is the transformation of declarative code into executable database logic. This process involves path-based resolution and schema awareness.
1. Definition Layer
Developers use defineTable and path builders (t.*) to describe the data structure. This layer is strictly type-safe; paths are validated against the database schema at compile time using template literal types.
export const usersTable = defineTable<typeof tables>()('users', (t) => ({
columns: [
t.text('name'),
t.text('profile.location'), // Dot-notation for relationship paths
t.text('posts.title'),
],
}));
Sources: README.md:43-52, plans/011-table-definition-dx-design.md:36-48
2. Adapter Translation
The adapter receives the FetchDataParams (including columns, filters, and sorting) and performs the following:
- Relationship Detection: Analyzes the schema to find foreign keys for dotted paths.
- JOIN Generation: Automatically constructs
LEFT JOINstatements only for the relationships required by the current active columns or filters. - SQL Emission: Converts the high-level logic into a dialect-specific query.
Sources: README.md:169-181, apps/marketing/src/components/home/pipeline.tsx:43-49
3. Filter and State Management
The TableStateManager (or individual managers for Filters, Sorting, and Pagination) handles the interactive state. In the "Contract v2" design, filters can be complex recursive trees (AND/OR groups) rather than simple flat arrays.
Key Contracts and Types
The pipeline relies on several core interfaces to maintain consistency across the boundary between @better-tables/core and the adapters.
FetchData Lifecycle
The primary interaction with the database occurs through the fetchData method, which is coordinated by the betterTables instance.
| Parameter | Type | Description |
|---|---|---|
columns | string[] | The list of active column IDs to project. |
filters | FilterNode | Boolean logic tree for data restriction (AND/OR). |
sorting | SortingState | Array of column IDs and directions. |
pagination | PaginationState | Current page and limit settings. |
Sources: plans/006-core-contract-v2-design.md:24-34, packages/core/src/types/table.ts:60-70
Interaction Sequence
The following sequence diagram shows how the BetterTable UI component communicates with the data layer.
Conclusion
The Data Flow & Pipeline in Better Tables abstracts the complexity of relational data management by leveraging a centralized definition that drives both query generation and UI state. By using a path-based type system and ORM-agnostic adapters, the system ensures that changes in the UI are accurately and efficiently reflected in the database layer while maintaining strict type safety throughout the lifecycle.
Sources: README.md:10-25, plans/011-table-definition-dx-design.md:24-34