Data Importer & Cutover Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• packages/workspace/src/store.ts
• plans/noesis-high-leverage-correction-plan.html
• plans/compounding-partnership-product-loop.html
• plans/noesis-autonomous-compounding-implementation-plan.html
• packages/workspace/src/importer.ts
• packages/domain/test/architecture.test.ts

The Data Importer and Cutover Strategy in Noesis facilitates a structural transition from a dual-store architecture—relying on JSONL journals and SQLite projections—to a single-authority model where SQLite is the primary source of truth for all operational data. This strategy is a core component of the "High-Leverage Correction Plan," designed to eliminate data divergence caused by the lack of transactional linking between disparate storage systems.

The scope of this strategy encompasses the migration of sessions, messages, tool calls, outcomes, jobs, and authority grants from legacy JSONL ledgers into the WorkspaceStore. Once the migration is verified, the system implements a versioned operational cutover protocol that halts dual-writing and redirects all foreground work to the unified SQLite-backed repositories.

Sources: plans/noesis-high-leverage-correction-plan.html:439-470, plans/compounding-partnership-product-loop.html:362-386

Cutover Architecture

The cutover process is designed to be atomic and recoverable. It utilizes a versioned protocol that ensures that if any part of the migration fails, JSONL operational writes remain active to prevent data loss. The WorkspaceStore manages this transition through a specific method, cutoverLegacyOperationalAuthority, which coordinates the backup, import, and marker update phases.

Core Components

• WorkspaceStore: Owns SQLite transactions, migrations, and the one-authority-per-datum rules.
• OperationalCutoverReport: A structured response indicating the status of the cutover, including legacy import details and source digests.
• ImportRun: A database record tracking the results of specific import attempts, including counts of migrated sessions and messages.
• AuthorityBoundary: Mediates the rehydration of receipt lineage and completion state from the migrated database rows.

Sources: packages/workspace/src/store.ts:60-72, packages/workspace/src/store.ts:658-675, plans/noesis-autonomous-compounding-implementation-plan.html:168-172

Cutover Execution Flow

The following diagram illustrates the sequence of operations performed during a legacy operational authority cutover.

This flow ensures that the system creates a verified backup before attempting to modify the authoritative operational state.

Sources: packages/workspace/src/store.ts:676-805, plans/noesis-high-leverage-correction-plan.html:455-470

Legacy Data Import Logic

The importer transforms chronological JSONL events into relational SQLite rows. It distinguishes between "Normal" (public), "Private", and "Secret" data sensitivity levels during the migration to maintain privacy boundaries.

Migrated Data Categories

Data Category	Target SQLite Table	Description
Sessions	sessions	Root interaction containers including metadata and status.
Messages	messages	Individual turns within a session, including roles and content.
Tool Calls	tool_calls	Records of tool execution, request payloads, and responses.
Outcomes	outcomes	Summaries and final statuses of turns or sessions.
Authority Grants	authority_grants	Persistent permissions granted to principals (e.g., users, agents).
Jobs	jobs	Background work items and their execution state.

Sources: packages/workspace/src/store.ts:708-800, packages/workspace/src/store.ts:894-910, plans/noesis-autonomous-compounding-implementation-plan.html:200-215

Conflict and Idempotency Handling

During the cutover, the importer enforces strict identity rules:

1. Grant Persistence: Grants are inserted using INSERT OR IGNORE based on the unique grant_id.
2. Operation Fingerprinting: For authority operations, the importer verifies that the operation_fingerprint in the legacy log matches any existing records. If the fingerprint has changed, the cutover is rejected.
3. Terminal State Immutability: Once a tool call or operation reaches a terminal state (completed/failed), its record becomes immutable.

Sources: packages/workspace/src/store.ts:717-730, packages/workspace/src/store.ts:756-775, packages/workspace/src/store.ts:930-945

Authority and Security Boundaries

The cutover strategy transitions the system to a "Protection by Construction" model. After cutover, all protected mutations require an AuthorityReceipt verified at the deep protected adapter level.

Single Authority Matrix

Following a successful cutover, the following authorities are established:

• SQLite: Authoritative for sessions, messages, tool calls, jobs, experiments, and activation pointers.
• Working Files: Authoritative for declarative material (prompts, skills, config, profile/memory).
• Revisions Area: Byte-stable snapshots of file revisions used for pinning and evaluation.

Sources: plans/noesis-autonomous-compounding-implementation-plan.html:200-215, packages/domain/test/architecture.test.ts:220-230

Receipt Verification Sequence

The deep adapter ensures that no component can modify the control plane without a verifiable receipt issued by the AuthorityBoundary.

Verification and Acceptance Criteria

The success of the cutover is measured by the complete retirement of the legacy journal for operational writes and the ability to rehydrate the full system state from SQLite.

Acceptance Checklist

• No Dual-Writes: Settlement of a turn produces no new JSONL appends.
• Idempotency: Re-running the cutover logic returns the alreadyCompleted flag without modifying data.
• Digest Integrity: The source_digest of the journal bytes must match the recorded marker to ensure the migration is based on a consistent snapshot.
• Failure Recovery: Interrupted migrations fail closed; the system reverts to JSONL authority until a successful cutover commit.
• Backup Verification: A backup located at backups/pre-operational-cutover-v1 must exist and be verifiable.

Sources: packages/workspace/src/store.ts:800-810, plans/noesis-high-leverage-correction-plan.html:455-470, plans/compounding-partnership-product-loop.html:420-435

Conclusion

The Data Importer and Cutover Strategy is the critical path for Noesis to achieve a transactional, unified storage model. By migrating legacy JSONL data to a versioned SQLite schema and implementing strict authority verification, the system eliminates the risks of storage divergence. This ensures that the "served plan" is always the "inspected plan," providing a trustworthy foundation for autonomous compounding and historical analysis.

Sources: plans/noesis-high-leverage-correction-plan.html:565-580