Agentic RAG & Knowledge Ingestion

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• apps/web/src/app/(app)/knowledge/page.tsx
• services/api/flow/interfaces/http/routes/knowledge.py
• services/api/scripts/seed_agents_and_datasets.py
• services/api/scripts/seed_skills.py
• services/api/tests/test_kg_parser.py

Agentic Retrieval-Augmented Generation (RAG) and Knowledge Ingestion form the backbone of Flow's capability to provide context-aware, cited answers. This system allows users to upload documents, crawl web content, and manage a semantic knowledge base that agents can query during execution. By leveraging vector databases and advanced LangGraph pipelines, the system transitions from simple retrieval to an "agentic" approach involving planning, grading, and multi-step reasoning.

The ingestion layer handles the transformation of unstructured data—ranging from PDFs and Word documents to raw markdown and web URLs—into searchable vector embeddings. Once ingested, these sources are partitioned into chunks and indexed within a retrieval corpus, enabling agents to cite specific evidence for their claims.

Knowledge Ingestion Architecture

Knowledge ingestion is the process of importing and processing data into the platform's long-term memory. The system supports multiple input formats and ensures that data is properly cleaned, chunked, and embedded.

Ingestion Flow

The following diagram illustrates the lifecycle of a document from upload to being ready for retrieval.

Supported Formats and Constraints

The ingestion engine enforces specific constraints to maintain performance and data quality.

Feature	Specification	Source
Max File Size	20 MB	knowledge.py:21
Allowed Extensions	.txt, .md, .mdx, .csv, .pdf, .docx	knowledge.py:22
Embedding Provider	OpenAI (API Key Required)	knowledge.py:44
Storage Engines	PostgreSQL (pgvector) & Qdrant	README.md:214-219

Extractor Logic

The system utilizes specialized extractors based on file suffixes:

• PDF: Uses extract_pdf to handle structured extraction of text and tables.
• DOCX: Uses extract_docx.
• Markdown/Text: Decoded as UTF-8 with error replacement.
• Web URLs: Uses extract_url_content which strips navigation and footers to index only the main article body.

Sources: services/api/flow/interfaces/http/routes/knowledge.py:25-38, README.md:61-63

Agentic RAG Pipeline

When Agentic RAG is enabled (FLOW_AGENTIC_RAG_ENABLED=true), retrieval shifts from a simple vector search to a complex LangGraph sub-pipeline.

Execution Flow

The standard retrieval flow follows a planner → worker → synthesizer pattern. In the agentic mode, the worker utilizes a more sophisticated retrieval strategy.

Retrieval Components

• Hybrid Search: Combines dense vector retrieval (pgvector) with Qdrant hybrid RRF + BM25 search.
• LLM Grader: Evaluates retrieved chunks for actual relevance to the query.
• Query Rewrite: If initial results are poor, the agent re-formulates the search query.
• Citations: Results are numbered [1]...[N] and emitted as Server-Sent Events (SSE) for the frontend to render.

Sources: README.md:126-130, README.md:214-219

Knowledge Management Interface

The web interface provides a dedicated space for managing the retrieval corpus through the /knowledge page.

Key Functionalities

• Source List: Displays all ingested sources with their title, creation date, chunk count, and ingestion status (indexed, processing, or failed).
• Chunk Preview: A drawer component allows developers to inspect the exact text chunks created from a document.
• Source Deletion: Allows removing sources from the workspace, which triggers a cleanup in the underlying vector stores.

Sources: apps/web/src/app/(app)/knowledge/page.tsx:32-90, services/api/flow/interfaces/http/routes/knowledge.py:165-174

Ingestion Statuses

Status	Description
indexed	Embedding and indexing are complete; available for RAG.
processing	Currently being chunked or sent to the embedding provider.
failed	An error occurred during extraction or embedding (e.g., invalid API key).

Sources: apps/web/src/app/(app)/knowledge/page.tsx:24-30

Knowledge Graph & Specialized Extraction

Beyond standard text retrieval, Flow supports structured knowledge extraction through specialized agent skills and knowledge graph parsing.

Specialized Extraction Skills

Agents like the Knowledge Curator are seeded with specific skills to extract structured information.

Skill Name	Purpose	Source
pdf-structured-extraction	Extracts tables, headings, and metadata into JSON.	seed_skills.py:46-51
knowledge-extraction	Identifies CONCEPT, PERSON, and ORG entities and their relationships (e.g., DEVELOPED_BY).	seed_skills.py:537-550
curation-deduplication	Merges overlapping knowledge entries to prevent redundancy.	seed_skills.py:603-610

Knowledge Graph Parsing

The system includes parsers for structured formats like Obsidian notes, enabling the construction of a Knowledge Graph.

• Frontmatter: Extracts YAML metadata (tags, dates).
• Wikilinks: Parses [[Link]] syntax to establish relationships between nodes.
• Tags: Extracts #tag identifiers from document bodies.

Sources: services/api/tests/test_kg_parser.py:1-40

Conclusion

Agentic RAG & Knowledge Ingestion in Flow represents a robust system for transforming raw data into actionable intelligence. By combining flexible ingestion routes, multi-engine vector storage, and an agentic retrieval pipeline involving LLM-based grading and rewriting, the system ensures high-quality, cited responses suitable for enterprise-grade assistant workflows.