File Artifacts & Storage
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The File Artifacts & Storage system within tldw_server manages the lifecycle of ingested media, generated outputs, and systemic artifacts. It provides a structured layer for persisting raw content (video, audio, documents), structured metadata in relational databases, and high-dimensional vector embeddings for retrieval-augmented generation (RAG) tasks.
The system is designed with a per-user isolation model, ensuring that content, notes, and vector stores are partitioned by user_id. This architecture supports both single_user and multi_user authentication modes, utilizing a combination of SQLite/PostgreSQL for metadata and ChromaDB for vector storage.
Sources: Docs/Architecture.md:46-52, AGENTS.md:58-70
Storage Architecture
The storage layer is bifurcated into centralized global storage and isolated per-user storage. Global storage handles system-wide data such as authentication, evaluations, and shared configurations, while per-user storage houses specific media assets and personal knowledge.
Storage Components Hierarchy
The system utilizes <USER_DB_BASE_DIR> (defaulting to Databases/user_databases/) to isolate user data.
Sources: Docs/Architecture.md:154-159, AGENTS.md:58-70
Database Management
Relational data is managed through specialized DB helpers located in core/DB_Management/. The project enforces a policy against raw SQL in endpoints, requiring all database interactions to use these abstractions.
| Database Type | Scope | Implementation | Purpose |
|---|---|---|---|
| AuthNZ | Global | SQLite/Postgres | User credentials, permissions, and session tokens. |
| Media DB v2 | Per-User | SQLite/Postgres | Media metadata, chunking status, and FTS5 search indexes. |
| ChaChaNotes | Per-User | SQLite | Personal notes, chat history, and character cards. |
| Evaluations | Global | SQLite | G-Eval metrics, RAG scores, and audit logs. |
| Vector Store | Per-User | ChromaDB | High-dimensional embeddings for semantic search. |
Sources: Docs/Architecture.md:161-193, AGENTS.md:58-70
File Ingestion and Artifact Lifecycle
Artifacts enter the system through the Ingestion Media Processing pipeline. This pipeline normalizes diverse formats into chunks and metadata, which are then persisted across the file system and database layers.
Ingestion Flow
- Client Upload: Media is received via
POST /api/v1/media/process-*endpoints. - Normalization: Modules use tools like
ffmpeg,yt-dlp, anddoclingto extract raw text or transcripts. - Persistence: Raw files are stored in the filesystem, while metadata is written to Media DB v2.
- Chunking: The
Chunkingmodule splits content based on configured strategies. - Vectorization: The
Embeddingsmodule generates vectors and stores them in ChromaDB.
Sandbox Artifacts & Sandbox Storage
Specialized artifacts are generated within the Sandbox subsystem, which handles isolated code execution. These artifacts have strict ownership and traversal guards.
- Sandbox Store: Manages execution state (runs, sessions) across
InMemoryStore,SQLiteStore, orPostgresStore. - Artifact Guards: The
SandboxArtifactGuardRouteand_require_run_ownerhelpers ensure that only the owner of a sandbox run can access generated files. - Traversal Protection: The system implements path validation to prevent directory traversal attacks during artifact downloads.
Sources: Docs/Plans/2026-03-08-sandbox-subsystem-review-implementation-plan.md:44-55, Docs/Plans/2026-03-08-sandbox-subsystem-review-implementation-plan.md:79-85
Workspace Artifact Management
The Workspace Playground redesign introduces a "Studio" concept for managing high-level artifacts such as summaries, mind maps, and flashcards.
Artifact Types
The system classifies generated outputs as GeneratedArtifact, tracking them with the following fields:
id: Unique identifier.type: Category (e.g.,summary,quiz,mindmap).status: Lifecycle state (pending,generating,completed,failed).serverId: Reference to the backend record inoutputs,quizzes, ordata-tables.
Sources: Docs/Product/WebUI/Workspace_Playground_Redesign.md:57-68, Docs/Product/WebUI/Workspace_Playground_Redesign.md:111-125
Data Flow for Studio Artifacts
Conclusion
The File Artifacts & Storage module provides the foundational infrastructure for data persistence and retrieval in tldw_server. By combining per-user filesystem isolation with structured relational databases and vector stores, the system ensures data privacy while enabling advanced AI-driven research capabilities. The integration of specialized components like the Sandbox Store and Workspace Artifacts further extends this capability to handle dynamic, generated content securely.