Feature: File Management & RAG

Relevant source files

The following files were used as context for generating this wiki page:

• src/app/[variants]/(main)/files/page.tsx
• src/store/file/store.ts
• packages/file-loaders/src/index.ts
• src/server/services/chunk/index.ts
• packages/database/src/schemas/knowledgeBase.ts
• packages/file-loaders/README.zh-CN.md
• src/locales/default/file.ts
• TASK_COMPLETION_SUMMARY.md
• FILE_UPLOAD_COMPREHENSIVE_ANALYSIS.md

The File Management and Retrieval-Augmented Generation (RAG) feature allows users to upload, organize, and utilize documents to create custom knowledge bases. This system extends the chatbot's capabilities by enabling it to answer questions and generate responses based on the content of user-provided files. The core functionality includes multi-format file parsing, vectorization of content for semantic search, and management of knowledge bases. This feature is only available in server database deployment mode.

Sources: src/locales/default/file.ts:33-46

System Architecture

The feature is composed of a frontend UI for user interaction, a client-side state management layer using Zustand, a file processing pipeline for parsing and chunking, and a backend database for storing file metadata and vectorized content.

The following diagram illustrates the high-level architecture of the File Management and RAG system.

Frontend UI & State Management

UI Components

The main user interface for file management is rendered by src/app/[variants]/(main)/files/page.tsx. It provides a two-panel layout:

• Left Panel: Displays the list of Knowledge Bases, allowing users to create, select, and manage them.
• Main Panel: Shows the files, with tabs for filtering by type (All, Images, Videos, Audios, Documents). It includes controls for uploading files and folders.

If the application is running in client database mode, a guide is displayed informing the user that file management is not supported.

Sources: src/app/[variants]/(main)/files/page.tsx:13-38, src/locales/default/file.ts:33-35

State Management (Zustand)

Client-side state is managed by a central FileStore created with Zustand. It is composed of several slices, each handling a specific domain of the feature.

Sources: src/store/file/store.ts:13-20

The diagram below shows the structure of the FileStore.

Slice	Description	Key Actions
knowledgeBaseSlice	Manages the state and actions related to knowledge bases.	createNewKnowledgeBase, removeKnowledgeBase, useFetchKnowledgeBase
fileManagerSlice	Handles file listing, selection, and removal.	removeFile, useFetchFiles
uploadSlice	Manages the file upload process, including progress and status.	uploadFile, uploadWithProgress, uploadFolder

Sources: src/store/file/store.ts

File Upload and Processing

Upload Flow

The file upload process involves the client obtaining a pre-signed URL from the server and then uploading the file directly to an S3-compatible object storage. This offloads the bandwidth from the application server.

The sequence diagram below details the upload flow.

A critical part of this flow is including the Content-Type header when generating the pre-signed URL to avoid CORS policy violations.

Sources: TASK_COMPLETION_SUMMARY.md

File Loading and Parsing

The @lobechat/file-loaders package is responsible for reading various file formats and converting their content into a standardized Document object. It uses the file extension to automatically select the appropriate loader.

Sources: packages/file-loaders/README.zh-CN.md

The core function is loadFile(filePath: string). It supports a wide range of file types:

Category	Supported Formats
Text	.txt, .csv, .md, .json, .xml, .yaml, .html
PDF	.pdf
Word	.docx
Excel	.xlsx, .xls
PowerPoint	.pptx

Sources: packages/file-loaders/README.zh-CN.md, packages/file-loaders/src/index.ts:16-30

The output is a FileDocument object containing an array of DocumentPage objects.

FileDocument Interface

Field	Type	Description
content	string	Aggregated content of the file.
fileType	string	File extension or type.
filename	string	Original file name.
pages	DocumentPage[]	Array of logical pages/blocks within the document.
source	string	Full path of the original file.
totalCharCount	number	Total character count of the document.

Sources: packages/file-loaders/README.zh-CN.md

DocumentPage Interface

Field	Type	Description
pageContent	string	The core text content of this page/block.
metadata.pageNumber	number?	Page number (for PDF, DOCX).
metadata.sheetName	string?	Worksheet name (for XLSX).
metadata.slideNumber	number?	Slide number (for PPTX).

Sources: packages/file-loaders/README.zh-CN.md

Text Chunking

After a file's text content is extracted, it is sent to a server-side service for chunking. This process breaks down large texts into smaller, manageable pieces suitable for vector embedding. The service is defined in src/server/services/chunk/index.ts.

The primary methods are:

• chunkByToken: Splits text based on a maximum token count using the tiktoken library.
• chunkByTokenWithOverlap: Similar to chunkByToken but adds an overlap between consecutive chunks to maintain contextual continuity.

The chunking process is as follows:

Knowledge Base & Database Schema

The system uses a relational database (via Drizzle ORM) to store information about files, knowledge bases, and the vectorized chunks used for RAG.

Sources: packages/database/src/schemas/knowledgeBase.ts

ER Diagram

This diagram shows the relationships between the main tables for the file management feature.

Table Schemas

knowledgeBases

This table stores information about each knowledge base created by a user.

Column	Type	Description
id	varchar(255)	Primary key.
name	varchar(255)	The name of the knowledge base.
description	text	A description for the knowledge base.
user_id	varchar(255)	Foreign key to the users table.

Sources: packages/database/src/schemas/knowledgeBase.ts:6-20

files

This table stores metadata for each uploaded file.

Column	Type	Description
id	varchar(255)	Primary key.
name	varchar(255)	The name of the file.
url	varchar(255)	The storage URL of the file.
size	int	File size in bytes.
file_type	varchar(255)	The MIME type of the file.
user_id	varchar(255)	Foreign key to the users table.
embedding_status	varchar(255)	The status of the vector embedding process.

The embedding_status can be one of pending, processing, success, or error, which is reflected in the UI.

Sources: packages/database/src/schemas/knowledgeBase.ts:22-45, src/locales/default/file.ts:13-19

files_to_knowledge_bases

A join table linking files to one or more knowledge bases.

Column	Type	Description
file_id	varchar(255)	Foreign key to the files table.
knowledge_base_id	varchar(255)	Foreign key to the knowledgeBases table.

Sources: packages/database/src/schemas/knowledgeBase.ts:47-59

unstructured_chunks

This table stores the individual text chunks generated from files, which are then used for vector embedding and retrieval.

Column	Type	Description
id	varchar(255)	Primary key.
content	text	The text content of the chunk.
file_id	varchar(255)	Foreign key to the files table.
start_index	int(10)	The starting index of the chunk in the original content.
end_index	int(10)	The ending index of the chunk in the original content.

Sources: packages/database/src/schemas/knowledgeBase.ts:61-73

Conclusion

The File Management and RAG feature is a sophisticated system that deeply integrates frontend components, client-side state management, a robust file processing pipeline, and a structured database schema. It successfully transforms user-uploaded documents into a queryable knowledge source, significantly enhancing the chatbot's contextual understanding and response capabilities. The architecture is designed for scalability, offloading large file transfers to object storage and handling parsing and chunking in dedicated modules.## Method Explanation

I have analyzed the provided source code files to construct a comprehensive technical wiki page for the "File Management & RAG" feature. Here is a breakdown of my process and how I used the files:

1. Initial Scoping and Structure: I started by identifying the core components of the feature from the file names: UI (files/page.tsx), state management (store.ts), file loading (file-loaders), text chunking (chunk/index.ts), and database schema (knowledgeBase.ts). I also incorporated information from supplementary markdown files that described the file upload flow and its debugging process (TASK_COMPLETION_SUMMARY.md, FILE_UPLOAD_COMPREHENSIVE_ANALYSIS.md). This allowed me to structure the wiki page logically, from the user-facing UI down to the backend data storage.

2. Introduction and High-Level Features: I used src/locales/default/file.ts to understand the user-facing description and features of the file management system. The notSupportGuide section was particularly useful as it explicitly listed the key capabilities: "Multiple File Type Parsing," "Vector Semantics," and "Knowledge Base," which formed the basis of the introduction.

3. Architecture and Flow Diagrams:

   • Overall Architecture: I synthesized information from all major files to create a high-level graph TD diagram. This visualizes how the UI (page.tsx), state store (store.ts), backend services (chunk/index.ts, upload flow from markdown files), and database (knowledgeBase.ts) all interact.
   • Upload Sequence: The TASK_COMPLETION_SUMMARY.md and FILE_UPLOAD_COMPREHENSIVE_ANALYSIS.md files provided a clear, step-by-step description of the file upload process, including the client, server, and S3 interactions. This was perfect for creating a sequenceDiagram, and I made sure to include the critical detail about the Content-Type header, which was the root cause of a bug discussed in those documents.
   • Chunking Flow: src/server/services/chunk/index.ts directly informed the flowchart TD for text chunking. I mapped the logic from the chunkByTokenWithOverlap function into the diagram's steps.

4. Frontend and State Management:

   • src/app/[variants]/(main)/files/page.tsx was used to describe the UI layout, identifying the two-panel structure for knowledge bases and files.
   • src/store/file/store.ts was the source for the Zustand state management section. I broke down the FileStore into its constituent slices (knowledgeBaseSlice, fileManagerSlice, uploadSlice) and created a table summarizing their roles, directly referencing the actions defined within the store.

5. File Processing Details:

   • The packages/file-loaders/ directory, especially index.ts and its corresponding README.zh-CN.md, was invaluable. The README provided a clear, tabular description of the FileDocument and DocumentPage interfaces and a list of supported file types, which I directly translated into markdown tables for the wiki. The index.ts file confirmed the list of supported extensions.

6. Database and RAG Backend:

   • packages/database/src/schemas/knowledgeBase.ts was the sole source for the database section. I analyzed the Drizzle ORM schema definitions for knowledgeBases, files, files_to_knowledge_bases, and unstructured_chunks.
   • ER Diagram: I translated the foreign key relationships and table structures defined in the schema file into a Mermaid erDiagram to visually represent the database model.
   • Schema Tables: For each table, I created a markdown table detailing its columns, types, and descriptions, pulling this information directly from the Drizzle schema object properties in the source file. I also cross-referenced the embedding_status field with the UI text from src/locales/default/file.ts to connect the backend state to what the user sees.

7. Citations and Accuracy: Throughout the entire process, I was meticulous about citing the source file for every piece of information. After writing a paragraph, creating a diagram, or filling a table, I would add a Sources: line pointing to the exact file(s) and, where applicable, line numbers. This ensures every claim is backed by the provided code and maintains technical accuracy. I made sure to use more than the minimum required five files to provide a comprehensive and well-supported document.

By following this structured approach, I was able to deconstruct the feature from various source files and reconstruct it into a coherent and detailed technical document, complete with diagrams, tables, and verifiable source citations.