Data Persistence Layers
Relevant source files
The following files were used as context for generating this wiki page:
Data persistence in OpenPoke is implemented through a decentralized architecture primarily utilizing flat-file storage and browser-based storage. The system manages three core types of persistent data: conversation transcripts, summarized "working memory," and operational state (such as seen emails and agent rosters). Most server-side data is stored within the server/data/ directory. Sources: README.md:84, server/services/conversation/log.py:16
1. Conversation Logs
The ConversationLog provides an append-only persistence mechanism for the interaction agent. It records every turn of the dialogue between the user, the interaction agent, and various execution agents. Sources: server/services/conversation/log.py:38-40
Architecture and Format
Messages are stored in a custom XML-like format on disk, including timestamps and HTML-encoded payloads. This structure allows the system to distinguish between different message types during retrieval. Sources: server/services/conversation/log.py:27-35
The diagram shows the flow of different message types into the persistent conversation log file. Sources: server/services/conversation/log.py:126-150
Supported Message Tags
| Tag | Description | Surfaces to User |
|---|---|---|
user_message | Content sent directly by the human user. | Yes |
agent_message | Status updates or results from execution agents. | No (Directly) |
poke_reply | The interaction agent's response to the user. | Yes |
wait | Orchestration metadata used to avoid redundant tasks. | No |
Sources: server/services/conversation/log.py:136-160
2. Working Memory and Summarization
The Working Memory layer acts as a long-term context buffer. It stores a condensed version of the conversation history to prevent LLM context window overflow while retaining critical information. Sources: server/services/conversation/summarization/working_memory_log.py:38-40
Data Structure
The working memory persists both a conversation_summary and a list of unsummarized_entries. The SummaryState object tracks the last_index of the conversation log that was processed, ensuring idempotency during summarization runs. Sources: server/services/conversation/summarization/working_memory_log.py:117-130
This class diagram represents the relationship between the persistence manager and the state objects used for summarization. Sources: server/services/conversation/summarization/working_memory_log.py:117-130, server/services/conversation/summarization/state.py
Persistence Logic
The system uses a temporary file replacement strategy (.tmp file) when writing summary states to ensure atomic updates and prevent file corruption during write failures. Sources: server/services/conversation/summarization/working_memory_log.py:146-155
3. Gmail Seen Store
To support proactive notifications via the ImportantEmailWatcher, the system maintains a persistent record of email IDs that have already been processed. Sources: server/services/gmail/importance_watcher.py:44-46
- Implementation: The
GmailSeenStoremanages a JSON-backed list of strings. - Limitation: It utilizes a
seen_limit(defaulting to 300) to prevent the tracking file from growing indefinitely. When the limit is reached, the oldest entries are truncated. - Storage Path: Typically stored at
server/data/gmail_seen.json.
Sources: server/services/gmail/importance_watcher.py:46-52, server/services/gmail/seen_store.py
4. Client-Side Persistence
The web interface utilizes browser localStorage to persist session-specific configuration and integration states. This allows the UI to maintain a consistent state across page refreshes without requiring a centralized database for user preferences. Sources: web/components/SettingsModal.tsx:14-25
Persisted Keys
| Key | Purpose |
|---|---|
user_timezone | The user's detected or manually overridden timezone. |
openpoke_user_id | A unique identifier (UUID) generated for the session. |
gmail_connected | Boolean flag indicating successful OAuth connection. |
gmail_email | The email address associated with the connected Gmail account. |
gmail_connection_request_id | The internal ID used to track the Composio OAuth flow. |
Sources: web/components/SettingsModal.tsx:142-160, web/components/SettingsModal.tsx:238-255
5. Summary Table of Server Files
| File Path | Persistence Role | Format |
|---|---|---|
server/data/conversation/poke_conversation.log | Raw dialogue history. | XML-Tagged Text |
server/data/conversation/poke_working_memory.log | Compressed context and summary. | XML-Tagged Text + JSON |
server/data/gmail_seen.json | Deduplication for email watcher. | JSON Array |
server/data/execution_agent.log | Internal tool execution journal. | Line-delimited Text |
Sources: server/services/conversation/log.py:17, server/services/conversation/summarization/working_memory_log.py:18, server/services/gmail/importance_watcher.py:41, server/services/execution/log_store.py
Conclusion
The Data Persistence Layers of OpenPoke favor simplicity and auditability by using human-readable flat files. By separating raw conversation logs from working memory summaries and local browser settings, the architecture ensures that the system remains lightweight while providing the necessary context for multi-agent orchestration and proactive user alerts.