Feature: Built-in Tools
Relevant source files
The following files were used as context for generating this wiki page:
Built-in tools extend the capabilities of LobeChat's AI assistants, allowing them to perform actions beyond generating text. These tools enable interaction with external systems, such as the user's local file system and the internet, or the creation of structured content like code and diagrams. The functionality is exposed to the Large Language Model (LLM) through detailed system prompts that define each tool's purpose, functions, parameters, and expected output format.
The system presents these capabilities to the user as "Built-in Plugins" or "Integrated Plugins". The core tools available are for local file system interaction, web browsing, and artifact creation. These tools are fundamental to creating powerful, pre-configured AI assistants known as "Bundled Apps" for specific use cases like code review or content writing.
Sources: src/tools/local-system/systemRole.ts, src/tools/web-browsing/systemRole.ts, src/tools/artifacts/systemRole.ts, locales/en-US/discover.json:20, [scripts/seedBundledApps/README.md:5-8]
Tool Architecture
The tool system is based on providing the LLM with a "system role" prompt that describes a set of available functions. The LLM interprets the user's request and, if appropriate, generates a response containing a structured command to invoke a tool. The application then parses this command, executes the corresponding function, and returns the result to the LLM to formulate a final response for the user.
This diagram illustrates the high-level data flow when a user prompt triggers a tool.
Sources: src/tools/local-system/systemRole.ts:1-2, src/tools/web-browsing/systemRole.ts:1-4, [src/tools/artifacts/systemRole.ts:1-2]
Local System Tool
This tool grants the assistant the ability to interact with the user's local file system. It is designed for tasks involving file management, such as listing directories, reading file contents, and organizing files. The system prompt provides the LLM with context about the user's operating system and paths to common directories like Desktop, Documents, and Downloads.
Sources: [src/tools/local-system/systemRole.ts:1-13]
Capabilities
The Local System tool exposes a set of functions for file system operations.
| Function | Description |
|---|---|
listLocalFiles | Lists files and directories in a specified path. |
readLocalFile | Reads the content of a specified file. Supports plain text, Word, Excel, PowerPoint, and PDF. Can read a specific line range. |
writeFile | Writes content to a specified plain text file (e.g., .txt, .md). |
searchLocalFiles | Searches for files based on keywords and other criteria like content, creation/modification date, and file type. |
renameLocalFile | Renames a single file or directory in its current location. |
moveLocalFiles | Moves one or more files or directories, with an option to rename during the move. |
Sources: [src/tools/local-system/systemRole.ts:16-22]
Workflow and Response Format
The LLM is instructed to select the appropriate tool and use the contextual paths if the user refers to a common location by name. A critical instruction is the response format: any reference to a local file or directory path in the assistant's response must be enclosed in a <localFile /> tag.
This diagram shows the workflow for reading a file using the Local System tool.
Sources: [src/tools/local-system/systemRole.ts:24-30, 71-85]
Web Browsing Tool
The Web Browsing tool provides the assistant with powerful internet access, enabling it to search across multiple search engines and extract content from web pages. This is used to provide users with up-to-date and comprehensive information.
Sources: [src/tools/web-browsing/systemRole.ts:1-4]
Capabilities
| Function | Description |
|---|---|
search | Searches the web using multiple search engines and categories. |
crawlMultiPages | Retrieves content from multiple web pages simultaneously. |
crawlSinglePage | Retrieves content from a specific web page. |
Sources: [src/tools/web-browsing/systemRole.ts:6-10]
Search Strategy and Syntax
The LLM is given detailed instructions on how to perform effective searches. This includes selecting appropriate search categories, engines, and time ranges based on the user's query.
A special syntax is used to control search behavior:
| Syntax | Description | Example |
|---|---|---|
!category | Specifies a search category. | !news latest AI developments |
!engine | Specifies a search engine. | !github react components |
:language_code | Specifies the search language. | :de !google bundestag |
site:domain.com | Restricts results to a specific website. | site:github.com SearXNG |
!time_range | Filters results by time. | !week AI safety |
Sources: [src/tools/web-browsing/systemRole.ts:27-51, 101-115]
The system is a metasearch engine that can leverage various sources:
| Category | Available Engines |
|---|---|
| General | google, bing, duckduckgo, brave, wikipedia |
| Academic | google scholar, arxiv |
| Code | github, npm, pypi |
| Videos | youtube, vimeo, bilibili |
| Images | unsplash, pinterest |
Sources: [src/tools/web-browsing/systemRole.ts:32-42, 89-100]
Citation Requirements
A strict requirement for the Web Browsing tool is to always cite sources using Markdown footnote format ([^1]) and list all referenced URLs at the end of the response.
Sources: [src/tools/web-browsing/systemRole.ts:53-69]
Artifacts Tool
The Artifacts tool allows the assistant to create substantial, self-contained content that is displayed in a separate UI window. This is intended for content that users might want to modify, reuse, or view in a rendered format, such as code, documents, diagrams, or even interactive components.
Sources: [src/tools/artifacts/systemRole.ts:1-3]
Artifact Creation Workflow
The assistant follows a specific workflow to generate an artifact:
- It first thinks in a
<lobeThinking>tag about whether the content is suitable for an artifact. - It then wraps the content in
<lobeArtifact>tags. - The opening tag must include
identifier,title, andtypeattributes.
This diagram shows the artifact creation process.
Sources: [src/tools/artifacts/systemRole.ts:24-31]
Artifact Types
The type attribute in the <lobeArtifact> tag specifies how the content should be handled and rendered.
| Type | Description | Attributes |
|---|---|---|
application/lobe.artifacts.code | For code snippets in any language. | language |
text/markdown | For plain text or Markdown documents. | |
text/html | For single-file HTML pages that can be rendered. | |
image/svg+xml | For SVG images that will be rendered. | |
application/lobe.artifacts.mermaid | For Mermaid diagrams that will be rendered. | |
application/lobe.artifacts.react | For React components that will be rendered. Supports recharts and lucide-react. |
Sources: [src/tools/artifacts/systemRole.ts:32-84]
Tool Manifest Structure
Under the hood, tools and plugins are defined by a manifest object. This manifest contains metadata, API definitions, and the system role prompt that instructs the LLM on its usage. The example below is for the midjourney plugin, which acts as a built-in tool.
| Manifest Key | Description |
|---|---|
identifier | A unique ID for the tool (e.g., "midjourney"). |
meta | Metadata including title, description, and avatar. |
api | An array defining the functions the tool exposes. Each function has a name, description, and parameters schema. |
systemRole | The detailed prompt that explains the tool's capabilities, rules, and structure to the LLM. |
settings | A schema for any required configuration, like API keys. |
Sources: [packages/database/src/repositories/dataImporter/tests/fixtures/with-client-id.json:5-120]
Here is a snippet of the manifest for the midjourney tool, showing how its API is defined.
{
"identifier": "midjourney",
"type": "plugin",
"manifest": {
"api": [
{
"name": "showMJ",
"parameters": {
"type": "object",
"required": ["prompts"],
"properties": {
"prompts": {
"type": "string",
"description": "发送给 mj 的 prompts"
}
}
},
"description": "当用户输入 /mj 时，唤起 MJ 插件面板。你需要生成 prompts 发送给 MidJourney"
}
],
"meta": {
"title": "Midjourney",
"description": "Unleash creativity with Midjourney..."
},
"systemRole": "As MidjourneyGPT, your role is to write, refine, and mix prompts for Midjourney..."
}
}
Sources: [packages/database/src/repositories/dataImporter/tests/fixtures/with-client-id.json:6-30, 56-57]