Vision & Image Generation
Relevant source files
The following files were used as context for generating this wiki page:
The Vision and Image Generation systems in Cogmo enable the agent to perceive multi-modal input and generate visual content. Vision is primarily handled through the LLM provider interface, allowing the agent to process documents (such as PDFs) and images, while Image Generation is implemented as a specialized capability leveraging external providers like fal.ai and Venice.ai.
These capabilities are integrated into the modular agentic loop, where tools can be used to trigger image creation, and the transport layer manages the delivery of resulting attachments to the user.
Sources: CLAUDE.md, design/integrations.md, src/setup/wizard.ts:397-402
Multi-Modal Vision Capabilities
Cogmo supports vision and document processing by translating multi-modal content blocks into provider-specific formats. This allows the agent to "see" images and read complex documents like PDFs as part of the conversation context.
Document and Image Processing
The system handles various media types by transcoding them into formats compatible with the LLM. For instance, the Anthropic provider translates document blocks into base64 or URL sources, supporting PDF and structured text formats (CSV, Markdown, HTML, JSON, XML, YAML).
Sources: src/llm/anthropic.test.ts:488-510, src/llm/anthropic.test.ts:528-540
Supported Media Types for Vision
| Category | Media Types | Implementation Notes |
|---|---|---|
| Binary Documents | application/pdf | Passed as base64 or URL source. |
| Structured Text | text/markdown, text/csv, text/html, application/json, application/xml, application/yaml | Transcoded through the text-source path. |
| Images | image/jpeg, image/png, image/gif, image/webp | Handled via ImageBlock in canonical types. |
Sources: src/llm/anthropic.test.ts:544-555, CLAUDE.md:144-147
The diagram above shows the data flow from a user file upload to the final LLM provider API call.
Sources: src/llm/anthropic.test.ts:488-560
Image Generation Architecture
Image generation is designed as a pluggable system using the Vercel AI SDK and specialized providers. It is exposed to the agent through a set of tools, allowing the assistant to generate visuals on demand based on user prompts.
Sources: CLAUDE.md:74-75, design/integrations.md:17-19
Provider Implementation
The system supports multiple backends for image generation, configured during the initial setup wizard:
- fal.ai: The primary provider for high-quality generations.
- Venice.ai: Supports native API features like
safe_mode(applying a blur) andnegative_prompt. - OpenAI-Compatible: Support for DALL-E or custom inference servers speaking the
/v1/images/generationsprotocol.
Sources: src/setup/wizard.ts:397-410, src/setup/wizard.ts:420-435
Configuration and Capabilities
Each image model is registered with specific capabilities that inform the agent's tool usage:
- Aspect Ratios: Supported ratios like
1:1,16:9, etc. - Seed: Support for deterministic generation.
- Negative Prompt: Ability to specify what not to include in the image.
- Image Input: Support for image-to-image or editing models.
Sources: src/setup/wizard.ts:518-545, src/setup/wizard.ts:555-565
The class diagram represents the relationship between providers, models, and their specific generation capabilities.
Sources: src/setup/wizard.ts:468-477, src/setup/wizard.ts:555-565
Setup and Validation
Configuration of Vision and Image Generation occurs through the cogmo setup wizard, which handles API key storage and provider validation.
Image Provider Setup Flow
- Provider Selection: Choose between Venice, fal.ai, or OpenAI-compatible.
- Credential Storage: API keys are stored as encrypted secrets in the
secretstable (AES-256-GCM). - Model Registration: Define model strings (e.g.,
flux-dev) and descriptions. - Capability Mapping: Set default attributes like
safe_modefor Venice.
Sources: src/setup/wizard.ts:410-440, src/setup/wizard.ts:470-490, README.md:65-68
Validation Logic
While LLM providers (Vision) are validated with a live /v1/models ping, image providers are saved without a live probe to avoid surprise billing for test generations. Errors are instead surfaced during the first actual generation attempt.
Sources: src/setup/wizard.ts:182-195, src/setup/wizard.ts:495-502
Self-Evolution and Vision
As part of the self-evolution ladder, the system can extract behavioural corrections related to how the agent handles visual data or generated images.
- Observation Extraction: The
ObserverInngest function inspects tool invocation history, includingtool_useblocks related to image generation. - Steering Rules: If a user corrects the agent's visual descriptions or image generation parameters, these are promoted to steering rules that are auto-injected into the system prompt.
Sources: design/evolution.md:9-15, design/evolution.md:162-168
Summary
The Vision & Image Generation system provides Cogmo with the ability to interact with the world visually. By abstracting provider-specific logic into clean interfaces (Anthropic, OpenAI, Venice, fal.ai), the agent can seamlessly process complex documents and generate high-quality images. The integration with the encrypted secrets store and the setup wizard ensures that these high-cost capabilities are managed securely and configured with appropriate model-specific constraints.
Sources: CLAUDE.md:144-150, src/setup/wizard.ts:518-530