Data Synchronization Engine

Relevant source files

The following files were used as context for generating this wiki page:

• server/common/sync/clonedown/index.ts
• server/common/sync/cloneup/index.ts
• server/common/sync/reconcile/index.ts
• server/common/sync/providers/dynamodb.provider.ts
• server/common/sync/providers/surreal.provider.ts
• shared/types/sync.type.ts
• server/common/sync/up/index.ts
• server/common/sync/down/index.ts
• server/common/sync/providers/index.ts
• server/common/sync/up/syncUpIntegration.test.ts

The Data Synchronization Engine is a core backend module responsible for maintaining data consistency between offline-first clients and the central server database. It provides a robust mechanism for clients to push their local changes (mutations) to the server and pull down the latest updates. The engine is designed with a pluggable provider architecture, allowing it to support different database backends like SurrealDB and AWS DynamoDB interchangeably.

This engine facilitates several key operations: uploading client changes (syncUp), downloading server changes (syncDown), initial data seeding (cloneUp and cloneDown), and data integrity checks (reconcile). Its primary goal is to enable a seamless offline-first user experience by reliably synchronizing data when a network connection is available.

Architecture

The synchronization engine is built around a provider-based pattern. A central SyncProviderFactory determines which database provider to use based on the SYNC_PROVIDER environment variable. All providers adhere to a common ISyncProvider interface, ensuring consistent functionality across different database systems.

This diagram illustrates the high-level architecture of the synchronization system.

Data Providers

The engine uses a factory to manage and instantiate data providers. This allows the application to switch between database backends via configuration without changing the core sync logic.

Provider Factory

The SyncProviderFactory is a singleton that returns the appropriate provider instance. It resets the provider instance if needed, which is particularly useful in testing environments.

ISyncProvider Interface

All data providers must implement the ISyncProvider interface, which defines the contract for all synchronization operations.

Method	Description
cloneUp	Handles bulk insertion of records for a given resource.
cloneDown	Fetches all records for a given resource.
syncUp	Processes a list of client mutations and applies them to the database.
syncDown	Fetches records modified after a specified timestamp for given resources.
reconcile	Compares client-side record checksums with server-side records to find inconsistencies.

Sources: server/common/sync/providers/index.ts:3-8

This class diagram shows the relationship between the interface and its concrete implementations.

Core Synchronization Processes

The engine exposes several endpoints to handle different synchronization scenarios.

syncUp

This is the primary process for sending incremental changes from the client to the server. The client batches local database mutations and sends them in the request body. The server-side provider then processes these mutations. After a successful syncUp, the server immediately triggers a syncDown process to send any new server-side changes back to the client.

The following sequence diagram shows the syncUp flow.

The syncUp function in the DynamoDB provider contains logic to handle mutations in batches of 25 to align with DynamoDB's BatchWriteItem limits.

Sources: server/common/sync/providers/dynamodb.provider.ts:101-118

cloneDown

This process is used to perform a full download of all records for a specific resource. It's typically used when a client needs to fetch a complete dataset for the first time.

• Request Body (ICloneDownBody): Contains the resource type to be cloned.
• Response: An array of all records for that resource.

Sources: server/common/sync/clonedown/index.ts:10-18, shared/types/sync.type.ts:10-12

cloneUp

This process allows for a bulk upload of records from the client to the server. This is useful for initial data migration or seeding the server database from a client's existing local data.

• Request Body (ICloneUpBody): Contains the resource type and an array of records to be uploaded.
• Response: A success or error message.

Sources: server/common/sync/cloneup/index.ts:10-18, shared/types/sync.type.ts:5-8

reconcile

The reconcile process is a data integrity check. The client sends a list of record IDs and their checksums (or versions) for a specific resource. The server compares these with its own records and returns a list of records that are missing on the client or have been updated on the server.

• Request Body (IReconcileBody): Contains the resource type and a records array of { id: string, modifiedAt: number }.
• Response: An array of full records that the client needs to update or create.

Sources: server/common/sync/reconcile/index.ts:11-20, shared/types/sync.type.ts:14-18

Key Data Structures

The synchronization processes rely on a set of well-defined types shared between the client and server.

Source: [shared/types/sync.type.ts]

Type	Fields	Description
ISyncUpBody	mutations: IMutation[] lastSyncDown: number resources: Resource[] dapId: string	The body for a syncUp request. Includes mutations, the timestamp of the last successful sync, and the resources the client is interested in.
IMutation	id: string dapId: string resource: Resource action: ResourceActionType resourceId: string | string[] params: any createdAt: number	Represents a single client-side database change (create, edit, delete).
ICloneUpBody	resource: Resource records: any[]	The body for a cloneUp request, containing records to be bulk-inserted.
ICloneDownBody	resource: Resource	The body for a cloneDown request, specifying which resource to download.
IReconcileBody	resource: Resource records: { id: string, modifiedAt: number }[]	The body for a reconcile request, containing client-side record metadata for comparison.

Summary

The Data Synchronization Engine is a critical component of the Nucleus architecture, enabling robust offline-first capabilities. Its modular, provider-based design allows for flexibility in the choice of backend database technology. By providing clear processes for incremental updates (syncUp), full data transfers (cloneUp/cloneDown), and integrity checks (reconcile), the engine ensures that user data remains consistent and available across all devices, regardless of network status. The integration tests confirm its functionality with both SurrealDB and DynamoDB, demonstrating its adaptability.

Sources: server/common/sync/up/syncUpIntegration.test.ts