Introduction to Flow
Relevant source files
The following files were used as context for generating this wiki page:
Flow is an open-source, highly observable personal "second brain" and agent platform designed to manage knowledge through document ingestion, autonomous agent execution, and continuous self-improvement loops. The system allows users to upload diverse document types, crawl web content, and interact with agents that provide cited answers based on a retrieved knowledge corpus.
The platform is built as a monorepo utilizing a modern high-performance stack, including FastAPI for the backend, LangGraph for agent orchestration, and Next.js for the user interface. It distinguishes itself through "metacognition" features, where agents improve themselves automatically via automated evaluations, prompt rewriting, and A/B testing of their configuration "genomes."
Sources: README.md:3-8, apps/web/src/components/marketing/HomeLanding.tsx:32-37
System Architecture
Flow employs a distributed architecture managed via Docker Compose, separating concerns between the user interface, API services, background processing, and data persistence.
High-Level Component Layout
- Frontend (
apps/web): A Next.js application using the App Router, Tailwind v4, and Shadcn-style UI primitives. It handles streaming responses via Server-Sent Events (SSE) and provides specialized views for knowledge management, agent configuration, and a knowledge graph. - Backend (
services/api): A Python-based FastAPI service (theflowpackage) containing the REST API, LangGraph agent definitions, and database migrations. - Background Workers: Powered by Redis and ARQ to handle asynchronous tasks such as document embedding, crawling, and the nightly continuous improvement loop.
- Persistence Layer: Uses PostgreSQL with
pgvectorfor structured data and semantic search, and optionally Qdrant for hybrid "agentic RAG" capabilities.
The following diagram illustrates the primary interaction between these services:
Local Infrastructure Configuration
For local development, the system exposes specific ports to the host machine to facilitate service access:
| Service | Host Port | Internal Port | Description |
|---|---|---|---|
| Web UI | 13000 | 3000 | Main user interface |
| API | 18000 | 8000 | FastAPI / OpenAPI documentation |
| Postgres | 55432 | 5432 | Database access |
| Redis | 16379 | 6379 | Task queue and caching |
| Qdrant | 16333 | 6333 | Optional vector search |
Sources: README.md:27-32, AGENTS.md:22-26
Knowledge Management and RAG
Knowledge in Flow is treated as a retrieval corpus used by agents to ground their responses. The ingestion pipeline supports multiple formats and entry points.
Ingestion Pipeline
Users can add knowledge via:
- File Uploads: Supporting PDF, .docx, .md, and .txt files up to 20MB.
- Web Crawling: Extracting and indexing main content from URLs while stripping navigation and footers.
- Manual Entry: Pasting raw text or markdown directly into the UI.
When content is ingested, it is chunked and embedded (requiring an OpenAI API key by default) before being stored in PostgreSQL using pgvector for cosine similarity searches.
Sources: README.md:41-55, apps/web/src/app/(app)/knowledge/page.tsx:320-350
Retrieval Execution
During an agent run, the system follows a specific LangGraph pipeline:
- Planner: Determines the approach to the user query.
- Worker: Executes retrieval (up to 8 chunks) and uses tools like long-term memory or sandboxed code execution.
- Synthesizer: Aggregates findings and emits a
citationsSSE frame.
Agent Genomes and Self-Improvement
Flow implements a "Genome" versioning system for agents, allowing for deterministic tracking and automated optimization of agent configurations.
Genome Structure
A genome is a versioned snapshot of an agent's logic, including:
- Template: The underlying graph structure (e.g.,
deer_flow). - System Prompt: Core instructions.
- LLM Config: Provider, model (e.g., GPT-4o), and temperature.
- Tools: Enabled capabilities like
retrieve,sandbox, orlong_term_memory. - Skills: Specific instruction blocks (YAML + XML markdown) that trigger based on intent.
The Continuous Improvement Loop
Flow uses an automated "Nightly ARQ" cron job (03:00 UTC) to evaluate and improve agents:
Data Modeling
The system's core state is persisted in PostgreSQL. The data model is designed to support multi-tenancy (workspaces), execution tracking, and agent memory.
| Table | Description |
|---|---|
agents | Stores agent definitions, templates, and links to configs. |
agent_skills | Versioned instructions with YAML frontmatter used by agents. |
executions | Tracks every run, its status (completed/failed), and metadata. |
episodic_memories | Stores agent memories with embeddings for later recall. |
agent_schedules | Configuration for recurring agent tasks (Cron). |
golden_results | Results from evaluation runs used for quality scoring. |
Sources: apps/web/src/app/(app)/dashboard/page.tsx:50-85, services/api/scripts/seed_skills.py:860-890
Summary
Flow provides a robust infrastructure for developers to build and maintain autonomous agents. By combining a "second brain" knowledge management system with a self-optimizing agent platform, it addresses the common challenges of LLM reliability and prompt engineering. The system remains highly observable through structured logging and its integrated Knowledge Graph, which maps executions and retrieved documents to identify semantic relationships.
Sources: README.md:120-130, apps/web/src/components/marketing/HomeLanding.tsx:15-25