Feature: Image Generation
Relevant source files
The following files were used as context for generating this wiki page:
The Image Generation feature provides users with the capability to create images from textual descriptions directly within the LobeChat application. This system is designed to be extensible, supporting multiple AI image generation providers like OpenAI's DALL·E 3, Google's Imagen, and others. It is integrated into the application as a dedicated tool, complete with a user interface for configuring generation parameters, managing tasks, and viewing results.
The feature is architected with a clear separation between the frontend UI, client-side state management, and backend services. This allows for a responsive user experience while handling potentially long-running image generation tasks asynchronously. The system also includes integration with the file management system, where generated images can be saved and organized.
Sources: packages/const/src/guide.ts:25-26, locales/en-US/image.json:62-65
Architecture Overview
The image generation process involves several components, from the user interface to the external AI provider. The flow begins on the client, which communicates with the backend via tRPC. The backend then orchestrates the call to the appropriate external image generation API.
The following diagram illustrates the high-level data flow for an image generation request.
Frontend Implementation
The frontend is responsible for rendering the user interface and managing the client-side state of image generation tasks.
Sources: src/app/[variants]/(main)/image/page.tsx
UI Component
The main user interface for this feature is provided by the Page component, located at src/app/[variants]/(main)/image/page.tsx. This is a client-side React component that allows users to input prompts, configure generation settings, and view the resulting images.
Sources: src/app/[variants]/(main)/image/page.tsx
State Management (Zustand)
Client-side state is managed by a Zustand store defined in src/store/image/store.ts. This store holds the image generation tasks, loading states, and configuration parameters.
Key properties of the ImageStore:
createImageTask: An action that initiates a new image generation task by calling the backend tRPC endpoint.imagesMap: A map storing the generated image data.taskQueue: An array of task IDs currently being processed.config: An object holding the current user-selected settings for generation (e.g., model, size, steps).
The diagram below shows the structure of the ImageStore.
User Configuration Options
The UI provides several options to control the image generation process. These labels are defined in the localization files.
| Configuration | Description |
|---|---|
Aspect Ratio | The aspect ratio of the generated image. |
Guidance Intensity | Controls how closely the image generation follows the prompt. |
Height | The height of the image in pixels. |
Width | The width of the image in pixels. |
Number of Images | The quantity of images to generate in a single task. |
Model | The AI model to use for generation. |
Seed | A number to ensure reproducibility. |
Size | Pre-defined resolution presets. |
Steps | The number of diffusion steps for the model to take. |
Sources: locales/en-US/image.json:2-40
Backend Implementation
The backend consists of a tRPC router for handling business logic and a Web API route for proxying requests to external providers.
tRPC Router
The tRPC router at src/server/routers/lambda/image.ts exposes endpoints for managing image tasks.
| Procedure | Description |
|---|---|
createImageTask | Creates a new image generation task, calls the text-to-image service, and saves it to the database. |
getTaskById | Retrieves a specific task by its ID. |
deleteTask | Deletes a task from the database. |
getAllTasks | Fetches all image generation tasks. |
Sources: src/server/routers/lambda/image.ts:13-39
Web API Endpoint
The API route src/app/(backend)/webapi/text-to-image/[provider]/route.ts acts as a proxy between LobeChat's backend and the external AI image providers. It is a dynamic route where [provider] can be openai, google, fal, etc.
This endpoint is responsible for:
- Authenticating the request.
- Retrieving the appropriate credentials for the specified provider.
- Forwarding the text-to-image request to the provider's API.
- Streaming the response back to the LobeChat client.
The sequence diagram below shows the flow for creating an image.
Tool Integration: DALL·E 3
LobeChat integrates with image generation models through a "tool" system. The DALL·E 3 integration is defined in src/tools/dalle/index.ts.
Tool Manifest
The DalleManifest provides metadata and the API definition for the tool.
identifier:lobe-image-designer. This unique ID is used to avoid conflicts with the name "dalle" in function calls.meta.title:DALL·E 3.api: An array defining the functions the tool exposes. For DALL·E 3, this istext2image.
Sources: src/tools/dalle/index.ts:6-10, src/tools/dalle/index.ts:49-53
text2image API Parameters
The text2image function accepts several parameters to control the image generation, defined by a JSON schema.
| Parameter | Type | Description | Required |
|---|---|---|---|
prompts | string[] | An array of up to 4 diverse prompts based on the user's request. | Yes |
quality | string | standard or hd. Defaults to standard. | No |
seeds | integer[] | A list of seeds to use for each prompt, useful for modifying previous images. | No |
size | string | Resolution of the image. Can be 1024x1024, 1792x1024 (wide), or 1024x1792 (tall). Defaults to 1024x1024. | No |
style | string | vivid (hyper-real) or natural. Defaults to vivid. | No |
Sources: src/tools/dalle/index.ts:11-47
System Role
The systemRole for the DALL·E 3 tool is a detailed set of instructions for the LLM on how to use the lobe-image-designer tool. Key instructions include:
- Prompt Generation: If the user doesn't specify a number, generate 4 diverse captions.
- Content Policy: Do not create images of public figures. Translate non-English descriptions.
- Prompt Formatting: Always start the prompt with the image type (e.g., "photo," "oil painting").
- Diversity: Explicitly include diverse descent and gender for any people depicted.
- Detailing: Prompts must be highly descriptive, detailed, and at least three sentences long.
Sources: src/tools/dalle/index.ts:54-95