Source Inventory Management

Relevant source files

The following files were used as context for generating this wiki page:

• src/source-inventory.ts
• src/source-inventory.test.ts
• src/types.ts
• src/status.ts
• src/indexer.ts
• src/selector.ts

Source Inventory Management is the subsystem within cxs responsible for scanning, analyzing, and reporting the state of raw Codex session files (JSONL) stored on the local file system. It serves as the bridge between the raw file storage (typically ~/.codex/sessions) and the internal SQLite index. Its primary purpose is to provide a "ground truth" of available source files to determine indexing coverage and freshness without requiring a full database read.

This system provides the metadata necessary for the CLI Status and Sync operations, enabling the application to detect when the index is stale or missing data for specific projects (working directories) or time ranges.

Sources: src/source-inventory.ts, AGENTS.md:38-40, README.md:1-5

Core Components and Data Structures

The management of source inventory relies on several key data structures that represent the state of the filesystem.

Source File Metadata

Each individual session file is represented by a SourceFileMeta object. This includes filesystem attributes (mtime, size) and extracted session metadata (CWD, date).

Field	Type	Description
filePath	string	The absolute path to the session JSONL file.
pathDate	string | null	The date extracted from the file path or name (ISO 8601).
cwd	string	The working directory associated with the session, read from the file header.
mtimeMs	number	Last modification time in milliseconds.
size	number	File size in bytes.

Sources: src/types.ts:66-72

Inventory and Snapshots

The system distinguishes between a general SourceInventory (a broad overview) and a SourceSnapshot (a filtered set of files corresponding to a specific Selector).

• SourceInventory: Aggregates files into cwdGroups and identifies the overall date range.
• SourceSnapshot: Contains a list of files matching a selector and a unique fingerprint used for freshness checks.

Sources: src/types.ts:54-64, src/source-inventory.ts:32-52

Architectural Logic and Data Flow

The inventory management process follows a top-down scanning approach, optimized to handle large directory trees without exhausting system resources.

Scanning Workflow

The sequence diagram below illustrates how collectSourceInventory processes the root directory to build the inventory.

CWD and Date Extraction

The system uses a fast-path scanner to extract metadata from raw .jsonl files. Instead of parsing the entire file, it reads the first 64KB (CWD_SCAN_BYTES) to find session_meta or turn_context events.

• Path Date Extraction: Matches patterns like YYYY/MM/DD or rollout-YYYY-MM-DD in the filename.
• CWD Extraction: Searches the file prefix for JSON lines containing a cwd field.

Sources: src/source-inventory.ts:68-75, src/source-inventory.ts:122-159

Fingerprinting and Freshness

A critical feature of the Source Inventory Management system is the generation of a deterministic fingerprint. This fingerprint is compared against the sourceFingerprint stored in the database's coverage records to determine if a re-sync is required.

Fingerprint Calculation

The fingerprint is a SHA-256 hash derived from the following attributes of all files in a snapshot:

1. Relative file path from the root.
2. Modification time (mtimeMs).
3. File size.
4. Extracted path date.
5. Extracted CWD.

Recommended Action Logic

The inventory system works with the status module to recommend user actions based on "freshness."

Status	Condition	Recommended Action
fresh	Snapshot fingerprint matches the DB coverage record.	query
stale	Files exist but metadata (mtime/size) has changed.	sync
missing	No coverage record exists for the given selector.	sync

Sources: src/status.ts:74-94, src/source-inventory.test.ts:36-54

Implementation Details

Optimized Directory Walking

The walkAsync function uses for await...of on directory handles to process entries sequentially. This prevents the "EMFILE: too many open files" error that occurs with unbounded concurrency (e.g., Promise.all) on large session directories.

// src/source-inventory.ts:89-105
for await (const entry of dirHandle) {
  const fullPath = `${currentDir}/${entry.name}`;
  if (entry.isDirectory()) {
    await walkAsync(fullPath, files, options);
    continue;
  }
  if (!entry.isFile() || !entry.name.endsWith(".jsonl")) continue;

  const stats = await stat(fullPath);
  const cwd = await readCwdMetadataAsync(fullPath, options);
  files.push({ filePath: fullPath, pathDate: extractPathDate(fullPath), cwd, mtimeMs: stats.mtimeMs, size: stats.size });
}

Sources: src/source-inventory.ts:89-110

Error Handling

The system employs a specific SourceInventoryError class to wrap filesystem failures, providing the path and the operation (e.g., "read directory", "stat file") that failed. During sync operations, errors can be suppressed using a bestEffort flag, though this prevents the writing of "complete" coverage records.

Sources: src/source-inventory.ts:18-27, src/indexer.ts:153-172

Summary

Source Inventory Management provides the essential foundation for cxs's progressive retrieval model. By performing lightweight filesystem scans and fingerprinting, it allows the system to validate index integrity and guide users through the status -> sync -> find workflow efficiently. It ensures that content answers always come from the index while maintaining a strict link to the ground truth of raw session files.