Data Persistence (Supabase Adapter)

Relevant source files

The following files were used as context for generating this wiki page:

• packages/adapter-supabase/src/supabase-persistence.ts
• packages/adapter-supabase/src/workflows/workflow-persistence.ts
• packages/adapter-supabase/src/evolution/evolution-persistence.ts
• packages/adapter-supabase/src/memory-persistence.ts
• packages/adapter-supabase/src/persistence-interface.ts
• packages/adapter-supabase/README.md
• packages/adapter-supabase/REFACTORING.md
• packages/adapter-supabase/src/nodes/node-persistence.ts
• packages/core/src/utils/persistence/memory/SupabaseStore.ts
• packages/core/src/utils/persistence/README.md

The @together/adapter-supabase package provides an optional persistence layer for the autonomous workflow system. It is designed to abstract the database interactions, allowing the core application to remain agnostic of the storage backend. The adapter supports both a live Supabase database for production and an in-memory alternative for testing and development, ensuring flexibility and testability.

This system is responsible for storing and retrieving all data related to workflow definitions, execution instances (invocations), agent (node) configurations, inter-agent messages, and the evolutionary history of workflows. It follows clean architecture principles, using dependency inversion and a factory pattern to decouple the core logic from the persistence implementation.

Sources: packages/adapter-supabase/README.md, packages/adapter-supabase/REFACTORING.md

Architecture

The persistence layer is built on a set of well-defined interfaces and modular, domain-specific classes. This design promotes separation of concerns and makes the system easier to maintain and extend.

Sources: packages/adapter-supabase/REFACTORING.md:21-36

Factory Pattern

A factory function, createPersistence, is the primary entry point for accessing the persistence layer. It allows consumers to select a backend (supabase or memory) at runtime, often controlled by the USE_MOCK_PERSISTENCE environment variable. This pattern is central to providing a swappable implementation for testing environments.

Sources: packages/adapter-supabase/README.md:13-22, packages/adapter-supabase/REFACTORING.md:10-18

import { createPersistence } from '@together/adapter-supabase'

// Use Supabase (requires environment variables)
const persistence = createPersistence({ backend: 'supabase' })

// Use in-memory (for tests)
const persistence = createPersistence({ backend: 'memory' })

// Auto-detect from USE_MOCK_PERSISTENCE env var
const persistence = createPersistence()

Sources: packages/adapter-supabase/README.md:16-22

Interface Segregation and Class Structure

The core IPersistence interface is segregated into smaller, more focused interfaces for different domains (INodePersistence, IMessagePersistence, IEvolutionPersistence). The SupabasePersistence class acts as a primary orchestrator, lazily initializing and delegating to specialized classes that handle the logic for each domain.

The diagram below illustrates the class structure and interface implementation.

Field Mapping

The adapter automatically handles the conversion between camelCase used in the application code and snake_case used in the Supabase database schema. This is handled by internal utility functions and is transparent to the consumer of the persistence layer.

Sources: packages/adapter-supabase/README.md:209-222, packages/adapter-supabase/REFACTORING.md:61-75

Database Schema

The Supabase backend relies on a relational schema to track workflows, their evolution, and their executions. The core tables are designed to capture the entire lifecycle and genealogy of agentic workflows.

The following Entity-Relationship diagram shows the primary tables and their relationships.

Key Tables

Table	Description
WorkflowVersion	Stores an immutable version of a workflow's configuration (dsl). Tracks genealogy through parent1_id, parent2_id, and generation_id.
WorkflowInvocation	Records a single execution of a WorkflowVersion. Tracks status, cost, and performance metrics like fitness_score.
NodeVersion	Stores the configuration of a specific node (agent) within a WorkflowVersion.
NodeInvocation	Records a single execution of a node within a WorkflowInvocation. Tracks cost, output, and summary.
EvolutionRun	Represents a complete evolutionary process with a specific goal.
Generation	Represents a single generation within an EvolutionRun, containing a population of WorkflowVersions.

Sources: packages/adapter-supabase/README.md:121-199

Core Persistence Components

Workflow Persistence (SupabaseWorkflowPersistence)

This component manages the lifecycle of workflows, including their versions and invocations.

• ensureWorkflowExists(workflowId, description): Creates or updates a workflow definition.
• createWorkflowVersion(data): Saves a new, immutable version of a workflow, including its configuration (dsl) and evolutionary context.
• createWorkflowInvocation(data): Creates a record for a new workflow run, setting its status to running.
• updateWorkflowInvocation(data): Updates an existing invocation with its final status, cost, and performance metrics.
• loadWorkflowConfig(workflowVersionId): Retrieves the dsl configuration for a given workflow version.

The following diagram shows the sequence for creating a new workflow version.

Node Persistence (SupabaseNodePersistence)

This component handles persistence for individual nodes (agents) within a workflow. It records both the node's configuration for a given workflow version and its execution details during an invocation.

A key pattern is the two-phase recording of a node's execution:

1. createNodeInvocationStart(data): Called at the beginning of a node's execution to create a record with status: 'running'. This returns a nodeInvocationId.
2. updateNodeInvocationEnd(data): Called at the end of execution to update the record with the final status (completed or failed), output, cost, and summary.

Other key methods include:

• saveNodeVersion(data): Saves a version of a node's configuration tied to a specific workflowVersionId.
• retrieveNodeSummaries(workflowInvocationId): Fetches all node summaries for a given workflow run, ordered by execution time.

Sources: packages/adapter-supabase/src/nodes/node-persistence.ts, packages/adapter-supabase/src/persistence-interface.ts:101-120

Evolution Persistence (SupabaseEvolutionPersistence)

This component is responsible for tracking the metadata of evolutionary processes. It is only used when the system is operating in a genetic programming or iterative evolution mode.

• createRun(data): Initializes a new evolution run, recording its goal and configuration. Returns a runId.
• createGeneration(data): Creates a new generation record within a specific run. Returns a generationId.
• completeGeneration(update, stats): Marks a generation as complete and records its performance statistics.
• getLastCompletedGeneration(runId): Retrieves the context of the most recent completed generation to resume an evolution run.

Sources: packages/adapter-supabase/src/evolution/evolution-persistence.ts

Alternative & Complementary Storage

In-Memory Persistence

The InMemoryPersistence class provides a complete, in-memory implementation of the IPersistence interface. Its primary purpose is to enable fast, isolated, and dependency-free testing without requiring a connection to a Supabase instance. It mimics the behavior of the Supabase adapter but stores all data in memory, which is cleared between test runs.

Sources: packages/adapter-supabase/src/memory-persistence.ts, packages/adapter-supabase/REFACTORING.md:124-132

Supabase Context Store

For storing large pieces of data, such as files or large context objects, the system uses SupabaseContextStore. This is distinct from the main database persistence and utilizes Supabase Storage (an S3-compatible object store).

Key features include:

• Scoped Storage: Data is stored in a path scoped by workflowInvocationId, e.g., {workflowInvocationId}/{scope}/{key}.
• Atomic Directories: For each key, it creates a "directory" containing the data, a summary, and metadata, which are uploaded together.
• Automatic Summarization: When data is stored, an AI-generated summary is created and saved alongside it in summary.txt.
• Retry Logic: Implements a retry mechanism with exponential backoff for network operations.

Sources: packages/core/src/utils/persistence/memory/SupabaseStore.ts

Error Handling

The adapter defines and uses a hierarchy of custom error types to allow for specific and type-safe error handling by the application. The base error is PersistenceError, which wraps the underlying database error.

Error	Description
PersistenceError	Base error for any failed persistence operation. Contains the original error in a cause property.
WorkflowNotFoundError	Thrown when a specific workflowVersionId cannot be found.
NodeVersionMissingError	Thrown when attempting to save a NodeInvocation for a NodeVersion that does not exist.
DatasetRecordNotFoundError	Thrown when one or more dataset records cannot be found.
InvalidInputError	Thrown when input data for a persistence operation is invalid (e.g., missing ID).

Sources: packages/adapter-supabase/README.md:173-191, packages/adapter-supabase/REFACTORING.md:38-59, packages/adapter-supabase/src/index.ts:30-36

Example of type-safe error handling:

import { WorkflowNotFoundError, PersistenceError } from '@together/adapter-supabase'

try {
  await persistence.loadWorkflowConfig('invalid_id')
} catch (error) {
  if (error instanceof WorkflowNotFoundError) {
    // Handle missing workflow
  } else if (error instanceof PersistenceError) {
    // Handle general persistence error
    console.error(error.cause)  // Access underlying error
  }
}

Sources: packages/adapter-supabase/README.md:180-191

Maintenance

The adapter includes a cleanupStaleRecords method to perform database maintenance. This function is designed to be run periodically to prevent the accumulation of orphaned records. Its tasks include:

• Updating the status of WorkflowInvocation and NodeInvocation records from running to failed if they are older than a defined threshold (e.g., 10 minutes).
• Updating the end_time for stale EvolutionRun and Generation records.
• Deleting old Message records.

Sources: packages/adapter-supabase/src/workflows/workflow-persistence.ts:286-353

Summary

The Supabase adapter is a critical component that provides a robust and flexible persistence solution for the lucky project. By abstracting database logic behind a clean interface and providing both a real and a mock implementation, it enhances the testability and maintainability of the entire system. Its domain-driven structure, custom error handling, and support for both relational data (Postgres) and object storage (Supabase Storage) make it a comprehensive data management layer for the complex needs of evolutionary agentic workflows.