Knowledge Management
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Knowledge Management within the tldw_server ecosystem is a multi-faceted domain responsible for the lifecycle of user-generated and AI-augmented information. Its primary purpose is to provide a unified environment for capturing, organizing, and interacting with notes, prompts, and research artifacts. This system serves as a "personal knowledge assistant," enabling users to synthesize ingested media into actionable insights through notebook-style interactions and automated processing pipelines.
The scope of Knowledge Management encompasses the Notes system, Prompt Management, and Chatbooks. These components are integrated with the core RAG (Retrieval-Augmented Generation) and LLM (Large Language Model) modules to facilitate sophisticated research workflows. Data is logically isolated per user through dedicated storage structures, ensuring privacy and scalability in both single-user and multi-tenant environments.
Sources: Architecture.md:40-50, AGENTS.md:55-65, CLAUDE.md:65-75
Architecture and Data Storage
Knowledge Management leverages a decentralized storage model where user content is logically isolated into per-user directories. The system distinguishes between "Content/Media" databases and "Notes/Chats" databases to optimize for different access patterns.
Storage Structure
The architecture utilizes a base directory for user databases, typically defaulting to Databases/user_databases/<user_id>/. Within this directory, several specialized SQLite databases manage the knowledge lifecycle:
| Database File | Purpose | Key Data Entities |
|---|---|---|
ChaChaNotes.db | Primary store for user interaction data. | Notes, Chat History, Character Data |
user_prompts_v2.sqlite | Centralized prompt and tool repository. | Projects, Prompts, Tests, Optimizations |
Media_DB_v2.db | High-volume media and metadata store. | Media items, chunks, metadata, FTS indexes |
chroma_storage/ | Vector-based knowledge representation. | Document embeddings and vector indices |
Sources: Architecture.md:200-240, AGENTS.md:140-160, CLAUDE.md:150-170
Knowledge Flow Diagram
The following diagram illustrates how raw information flows from ingestion through to the various Knowledge Management storage layers and interaction modules.
Sources: Architecture.md:165-185, AGENTS.md:195-210
Notes and Notebook System
The Notes system provides a notebook-style environment for creating and managing research artifacts. It is designed to be highly interoperable, supporting various import and export formats to bridge the gap between external tools (like Google Keep) and the tldw_server internal knowledge base.
Key Capabilities
- Import/Export: Supports
.json,.md, and.markdownfiles. - Title Extraction: Derived automatically from Markdown headers (
# Heading), filenames, or the first non-empty line of content. - Metadata Integration: Markdown front matter can populate
keywords:ortags:within the system. - Duplicate Strategies: When importing, the system handles collisions via
skip,overwrite, orcreate_copystrategies.
Sources: 2026-03-07-google-keep-notes-import-export-design.md:30-50, 2026-03-07-google-keep-notes-import-export-implementation-plan.md:100-120
External Integration: Google Keep
The system includes specialized workflows for migrating data from Google Keep. This involves a multi-step process:
- Extraction: Utilizing Google Takeout to export raw Keep data.
- Conversion: Using the
keep-it-markdownbridge to transform Takeout HTML/JSON into tldw-compatible Markdown. - Ingestion: Uploading Markdown files to the
POST /api/v1/notes/importendpoint.
Sources: 2026-03-07-google-keep-notes-import-export-design.md:65-80
Prompt Management and Studio
The Prompt Studio is a core component of the knowledge management suite, allowing users to manage complex instruction sets for AI interactions. It is structured into projects that contain prompts, associated tests, and optimization parameters.
Prompt Lifecycle Sequence
Sources: AGENTS.md:105-110, Architecture.md:150-160
Chatbooks and Artifact Generation
Chatbooks represent a high-level abstraction for exporting and importing grouped knowledge artifacts. They allow for the persistence of entire research contexts, including chat histories and related media references, for background job handling or archival purposes.
Writing and Generation Assistance
The system includes a dedicated Writing module that provides specialized tools for knowledge synthesis:
- Drafting and Outlining: Automating the initial structure of research papers or notes.
- Rewriting and Formatting: Transforming raw transcriptions or notes into polished documents.
- Contextual Generation: Utilizing prompt/contexts and RAG results to generate high-fidelity text.
Sources: Writing/README.md:5-20, AGENTS.md:110-115
Summary
Knowledge Management in tldw_server effectively bridges raw data ingestion and intelligent information retrieval. By leveraging per-user SQLite databases for notes, prompts, and chats, the system ensures that user knowledge is persistent, searchable, and ready for AI-assisted synthesis. The architecture's emphasis on thin endpoints and logic-heavy core modules allows for complex operations like hierarchical chunking and multi-format note migration while maintaining high performance and data integrity.
Sources: Architecture.md:270-285, AGENTS.md:345-360