Local JSON File Database

Relevant source files

The following files were used as context for generating this wiki page:

• packages/db/src/store.ts
• packages/db/src/paths.ts
• packages/db/src/reset.ts
• CONTRIBUTING.md
• apps/desktop/README.md
• services/agent/src/tools.ts

The Local JSON File Database is a lightweight, pure-JavaScript storage system designed to replace SQLite in the OpenLive project. It serves as a single-user data store that maintains persistence for keys, settings, and conversations without requiring native modules, ensuring compatibility across Electron desktop builds and containerized environments.

This system is structured around a "table-per-file" architecture where each data category is stored in its own JSON file. It is optimized for low-write rates and high reliability through atomic write operations, allowing multiple processes (such as the web UI and the agent) to interact with the data safely.

Sources: packages/db/src/store.ts:4-13, CONTRIBUTING.md:27

Architecture and Design Principles

The database follows a minimalist design aimed at maintaining a small codebase and avoiding the complexity of native database drivers.

Atomic Writes and Process Safety

To prevent data corruption and race conditions, the system employs an atomic write strategy. When writing data, the system first creates a temporary file associated with the current process ID. Once the write is successful, it performs a synchronous rename operation to replace the target file. Since renames are atomic on the same filesystem, readers in other processes never encounter partially written or corrupted files.

The system assumes a single writer per "table" to avoid write-write races: the Web process writes providers and settings, while the Agent process writes conversation data.

Sources: packages/db/src/store.ts:8-13, packages/db/src/store.ts:25-30

Storage Locations

Data is stored relative to the project root or an environment-specified directory.

Directory Variable	Path Logic	Description
REPO_ROOT	../../.. from packages/db/src	The root of the repository.
DATA_DIR	process.env.OPENLIVE_DATA_DIR or REPO_ROOT/data	Primary storage for JSON files.
SCRATCH_DIR	DATA_DIR/scratch	Temporary or volatile data storage.

Sources: packages/db/src/paths.ts:5-12

Core Components and Data Flow

The database functionality is encapsulated in a few key utility functions and scripts that manage the lifecycle of the JSON files.

Utility Functions

The store.ts module provides the primary interface for data access:

• readJson<T>(name: string, fallback: T): T: Reads a file synchronously from disk and parses it as JSON. If the file is missing or unreadable, it returns the provided fallback value.
• writeJson(name: string, data: unknown): void: Ensures the DATA_DIR exists, then performs an atomic write of the provided data to the specified filename.

Sources: packages/db/src/store.ts:17-30

Data Schemas

The database manages several key files, primarily used for configuration and persistence:

File Name	Purpose	Managed By
settings.json	General application settings and preferences.	Web Process
providers.json	API keys and configuration for AI providers.	Web Process
conversations.json	History of user interactions and agent turns.	Agent Process

Sources: packages/db/src/reset.ts:6, packages/db/src/store.ts:8-10

Reset Mechanism

The system includes a reset script that allows for clearing all local data by recursively deleting the primary JSON files and the scratch directory.

// From packages/db/src/reset.ts
for (const p of ["providers.json", "settings.json", "conversations.json"]
    .map((f) => resolve(DATA_DIR, f))
    .concat(SCRATCH_DIR)) {
  if (existsSync(p)) rmSync(p, { recursive: true, force: true });
}

Sources: packages/db/src/reset.ts:6-10

Integration with Agent Tools

The local database is directly utilized by the agent's tool system for persistent memory. Specifically, the remember tool uses the database to store facts across turns.

Agent Notes Data Flow

The remember tool appends short facts to an agent_notes setting, which is then injected into the system prompt for subsequent interactions.

Conclusion

The Local JSON File Database provides a robust yet simple persistence layer for OpenLive. By leveraging atomic filesystem operations and a pure-JavaScript implementation, it maintains high performance for single-user scenarios while ensuring the application remains easy to build and distribute across different platforms without native dependency overhead.

Sources: packages/db/src/store.ts:4-13, apps/desktop/README.md:12-14