EFS Persistence & Logging

Relevant source files

The following files were used as context for generating this wiki page:

• src/persistence/context.ts
• src/log.ts
• src/persistence/memory.ts
• deploy/cloudformation.yml
• entrypoint.sh
• src/agent/setup.ts
• CLAUDE.md

The pi-digby system utilizes Amazon Elastic File System (EFS) to provide a persistent storage layer for an otherwise ephemeral Docker environment running on AWS ECS Fargate. This system ensures that critical agent state, including conversation history, long-term memory, and custom tool definitions, survives container restarts and deployments.

Persistence is achieved by mounting an EFS volume at /data within the container. This directory serves as the root for all persistent artifacts, including channel-specific logs, global configuration files, and indexed vector embeddings used for retrieval.

Sources: CLAUDE.md:10-15, deploy/cloudformation.yml:430-440

Infrastructure & Mount Strategy

The storage architecture relies on an AWS EFS FileSystem configured with a POSIX access point to ensure consistent UIDs and GIDs. The container's entrypoint script manages the symbolic linking of application-specific configuration directories to the persistent mount point.

Workspace Layout

The persistent storage at /data is organized into a hierarchical structure to separate global state from channel-specific context.

Path	Purpose
/data/.pi	Global configuration, including mcp.json and authentication tokens.
/data/MEMORY.md	Global workspace memory shared across all channels.
/data/skills/	Global CLI tools (skills) available to the agent.
/data/<channelId>/	Per-channel directory containing localized state.
/data/<channelId>/log.jsonl	Append-only message history for the specific channel.
/data/<channelId>/context.jsonl	Session context used by the pi-agent-core SessionManager.
/data/<channelId>/MEMORY.md	Channel-specific long-term memory.
/data/.cache/qmd	SQLite database and embeddings for vector-based search.

Sources: src/agent/prompt.ts:64-75, entrypoint.sh:23-35, src/persistence/memory.ts:15-35

Logging Architecture

The system maintains two distinct types of logs: Structured Application Logs (stdout/stderr) and Persistent Conversation Logs (log.jsonl).

Structured Application Logging

The application uses a centralized logger to output timestamped, structured information to stdout. This output is captured by AWS CloudWatch via the awslogs driver.

Level	Function	Description
Info	log.info	General system events and startup status.
Warn	log.warn	Recoverable errors, retries, or socket reconnections.
Error	log.error	Critical failures.
Tool	log.toolStart/End	Tracks execution of agent tools, including duration and status.
Usage	log.usage	Summarizes token usage, cost, and context window utilization.

Sources: src/log.ts:3-43, deploy/cloudformation.yml:460-465

Conversation History (log.jsonl)

Every channel contains a log.jsonl file. This file acts as a permanent record of user messages and bot responses, formatted as JSON lines. It is used to backfill history when the bot is offline and provides searchable context for the agent through the grep tool.

Memory & Context Persistence

The agent's "memory" is reconstituted at the start of every run by reading from multiple persistent files on EFS.

Context Synchronization

On startup or per-message run, the system synchronizes the append-only log.jsonl with the context.jsonl file used by the underlying SessionManager. This ensures that messages received while the harness was inactive are integrated into the agent's current conversation session.

// From src/agent/setup.ts
const syncedCount = syncLogToContext(sessionManager, channelDir, event.ts, event.threadTs);
if (syncedCount > 0) {
    log.info(`[${channelId}] Synced ${syncedCount} messages from log.jsonl`);
}

Sources: src/agent/setup.ts:210-215, src/persistence/context.ts

Memory Loading Logic

The loadMemory function aggregates global and channel-specific MEMORY.md files into a single string for the system prompt.

1. Global Memory: Located at /data/MEMORY.md. Contains preferences and project-wide info.
2. Channel Memory: Located at /data/<channelId>/MEMORY.md. Contains channel-specific decisions.

Sources: src/persistence/memory.ts:15-38, src/agent/prompt.ts:145-150

Vector Search & Embeddings (QMD)

The entrypoint.sh script initializes a vector storage system (QMD) on the EFS volume. It registers three collections:

• memory: Global workspace notes.
• channels: Per-channel memory and summaries (masked to */memory/**/*.md).
• skills: Definitions of reusable CLI tools (**/skills/**/SKILL.md).

The system runs a background process (qmd update && qmd embed) to ensure the SQLite-backed index on EFS is up to date with the files stored in the persistent volume.

Sources: entrypoint.sh:34-48

Data Retention & Backup

AWS Backup is configured via CloudFormation to perform daily snapshots of the EFS volume.

• Schedule: Daily at 03:00 UTC.
• Retention: 5 days.
• Lifecycle: Transition to Infrequent Access (IA) storage after 30 days for cost optimization.

Sources: deploy/cloudformation.yml:115-117, deploy/cloudformation.yml:505-520