Storage & Arweave Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Storage & Arweave Integration system provides a decentralized and permanent storage layer for the Recoup API. Its primary purpose is to handle the persistence of text-based content, artist knowledge bases, and metadata on the Arweave blockchain, ensuring that critical artist data and generated reports remain accessible and immutable.
This system bridges the gap between the application's business logic and the Permaweb, providing utilities for uploading raw text, generating structured metadata JSON, and retrieving content through Arweave gateways. It is a core component used by AI agents to store campaign briefs, release plans, and research reports.
Sources: lib/files/generateAndStoreTxtFile.ts:1-10, CLAUDE.md:21-27, lib/mcp/tools/files/registerGenerateTxtFileTool.ts:26-30
Architecture Overview
The integration follows a tiered storage approach. While operational data is stored in Supabase, permanent assets and large text blocks are offloaded to Arweave. The system uses the arweave NPM package to interface with the network.
Sources: package.json:54, CLAUDE.md:41-45
Core Components
- Arweave Client: Manages the connection to the Arweave network and handles transaction signing.
- Storage Handlers: Specialized functions like
generateAndStoreTxtFilethat orchestrate the upload of both raw data and its corresponding metadata. - Gateway Utilities: Functions that transform native Arweave URI schemes (e.g.,
ar://) into fetchable HTTPS URLs for frontend or AI consumption. - Database Sync: Logic that links Arweave transaction IDs to Supabase records (e.g., in the
account_infotable).
Storage Data Flow
The following diagram illustrates the process of generating and storing a text-based asset on Arweave:
The diagram shows the dual-upload strategy where the raw text is uploaded first, followed by a metadata file that references the original content.
Sources: lib/files/generateAndStoreTxtFile.ts:20-63
File Generation and Persistence
The system provides a standardized way to convert application-generated text into permanent on-chain files. This is primarily handled by the generateAndStoreTxtFile function.
generateAndStoreTxtFile
This function performs three main tasks:
- Raw Upload: Uploads the provided string to Arweave as
text/plain. - Metadata Creation: Uploads a JSON object to Arweave containing the file's description, name (truncated to 100 characters), and an
animation_urlpointing to the raw text file. - Base64 Encoding: Returns a base64 version of the content for immediate local use in the API response.
Sources: lib/files/generateAndStoreTxtFile.ts:20-65
Metadata Structure
The metadata follows a standard NFT-like schema to ensure compatibility with various blockchain explorers and tools:
| Field | Type | Description |
|---|---|---|
image | String | Fixed placeholder icon (ar://EXwe2...) |
animation_url | String | The Arweave URL of the raw text file |
content | Object | Includes mime type (text/plain) and uri |
description | String | The full text content |
name | String | First 100 characters of the content |
Sources: lib/files/generateAndStoreTxtFile.ts:40-55
Artist Knowledge Base Integration
One of the primary use cases for Arweave storage is the Artist Knowledge Base. This allows AI agents to "learn" about an artist by reading text files stored on-chain.
createKnowledgeBase
This function integrates Arweave storage with the Supabase database. It uploads text content to Arweave and then appends the resulting metadata to the knowledges array in the account_info table.
The flow ensures that knowledge base entries are unique by URL and persist both in the decentralized storage and the local relational database.
Sources: lib/artist/createKnowledgeBase.ts:16-64
Retrieval Logic
When the AI agent needs to access the knowledge base, the system fetches content from the URLs stored in the knowledges array. The system specifically filters for text-based MIME types (text/plain, text/markdown, application/json, text/csv) and excludes binary files like images or PDFs.
Sources: lib/files/tests/getKnowledgeBaseText.test.ts:40-100
External Interfaces (MCP Tools)
The Arweave storage system is exposed to AI agents via the Model Context Protocol (MCP).
generate_txt_file Tool
The generate_txt_file tool allows agents to trigger the storage pipeline directly. It handles error reporting for common issues like missing API keys or content policy violations.
- Input:
contents(String, required) - Output: A success object containing the
arweaveUrlandmetadataArweaveURL, both processed through a fetchable gateway.
Sources: lib/mcp/tools/files/registerGenerateTxtFileTool.ts:15-45
Data Models
Knowledge Object
The standard data structure for an Arweave-linked asset in the database:
| Property | Type | Description |
|---|---|---|
url | String | The fetchable HTTPS URL via Arweave gateway |
name | String | Descriptive name (often the first line of text) |
type | String | MIME type (defaults to text/plain) |
Sources: lib/artist/createKnowledgeBase.ts:31-35
GeneratedTxtResponse
The response shape returned by the storage utility:
| Property | Type | Description |
|---|---|---|
txt.base64Data | String | Content encoded in base64 |
txt.mimeType | String | Always text/plain |
arweave | String | Raw Arweave URL for the text file |
metadataArweave | String | Arweave URL for the metadata JSON |
Sources: lib/files/generateAndStoreTxtFile.ts:4-10
Summary
The Arweave integration provides a robust, decentralized foundation for the Recoup API's long-term memory. By combining on-chain persistence with local metadata tracking in Supabase, the system enables AI agents to generate, store, and retrieve artist-specific knowledge safely and permanently. Key components like generateAndStoreTxtFile and createKnowledgeBase ensure that text content is not only saved but also enriched with searchable metadata.