Vision & Media Analysis

Relevant source files

The following files were used as context for generating this wiki page:

• packages/scoutline/src/commands/vision.ts
• packages/scoutline/src/lib/api-client.ts
• packages/scoutline/src/providers/zai/adapter.ts
• packages/scoutline/src/providers/zai/media.ts
• packages/scoutline/src/lib/image.ts
• packages/scoutline/src/providers/minimax/vision-mappings.generated.ts

Vision & Media Analysis provides multimodal source investigation capabilities for analyzing images, PDFs, and video content. The system uses a provider-agnostic command layer that dispatches requests to specialized AI adapters, primarily supporting Z.AI and MiniMax providers.

Sources: packages/scoutline/src/commands/vision.ts:80-141, packages/scoutline/package.json:4

System Architecture

The vision system follows a decoupled architecture where commands define the intent and instruction, while provider-specific adapters handle media constraints and API communication.

Data Flow Overview

1. Command Invocation: The user executes a vision subcommand (e.g., analyze).
2. Request Construction: The command layer builds a VisionRequest containing the source path, instructions, and operation type.
3. Provider Selection: The dispatcher selects an effective provider (defaulting to zai).
4. Media Validation: Provider-specific media modules validate file existence, extensions, and size limits.
5. Execution: The ZaiApiClient or corresponding provider client sends multimodal messages to the remote API.

The diagram shows the progression from user input through validation and provider dispatch to the final API response.

Sources: packages/scoutline/src/commands/vision.ts:16-35, packages/scoutline/src/providers/zai/adapter.ts:607-680

Vision Commands and Operations

The CLI exposes several specialized subcommands for different visual analysis tasks.

Command	Operation ID	Description	Provider Support
analyze	interpret-image	General image interpretation	Z.AI + MiniMax
ui-to-code	ui-artifact	Convert UI screenshot to code	Z.AI + MiniMax
extract-text	extract-text	OCR for code and documents	Z.AI + MiniMax
diagnose-error	diagnose-error	Analyze error screenshots	Z.AI + MiniMax
diagram	diagram	Interpret technical diagrams	Z.AI + MiniMax
chart	chart	Analyze data visualizations	Z.AI + MiniMax
diff	diff	Compare two UI screenshots	Z.AI only
video	video	Analyze video content	Z.AI only
batch	N/A	Process multiple inputs	Shared Runner

Sources: packages/scoutline/src/commands/vision.ts:80-141, packages/scoutline/src/providers/zai/adapter.ts:98-109

Media Constraints and Validation

Media analysis enforces strict limits on file sizes and formats to ensure compatibility with backend providers.

Provider-Specific Limits

Feature	Z.AI (General)	Z.AI (OCR)	MiniMax
Images	<= 5 MiB	<= 10 MiB	<= 50 MiB
PDFs	Not Supported	<= 50 MiB	Not Supported
Videos	<= 8 MiB	Not Supported	Not Supported
Formats	JPG, JPEG, PNG	JPG, JPEG, PNG, PDF	JPG, JPEG, PNG, WebP

Sources: packages/scoutline/src/providers/zai/media.ts:43-60, packages/scoutline/src/commands/vision.ts:128-132

Z.AI URL Fallback Mechanism

The Z.AI adapter implements a fallback mechanism for unreliable server-side URL fetching. If the provider's server-side fetcher fails, the adapter downloads the URL to a local temporary file and retries the request using the local path.

The sequence illustrates how the system recovers from server-side URL resolution failures by switching to a local file upload.

Sources: packages/scoutline/src/providers/zai/adapter.ts:696-725, packages/scoutline/src/providers/zai/media.ts:187-200

API Communication

The ZaiApiClient manages multimodal message construction. It supports system, user, and assistant roles with content types for text, images, and videos.

Multimodal Message Structure

// From packages/scoutline/src/lib/api-client.ts
export type MessageContent = 
  | { type: "image_url"; image_url: { url: string } }
  | { type: "video_url"; video_url: { url: string } }
  | { type: "text"; text: string };

export interface Message {
  role: "system" | "user" | "assistant";
  content: string | MessageContent[];
}

For models like glm-5.2 and above, the client automatically enables a reasoning_effort: "max" parameter and sets thinking: { type: "enabled" } to optimize analysis quality.

Sources: packages/scoutline/src/lib/api-client.ts:10-38, packages/scoutline/src/lib/api-client.ts:153-176

Batch Processing

The vision batch subcommand processes multiple visual media files using either a glob pattern or a JSON manifest.

Glob Expansion Rules

1. Single Directory: Globs must target a single directory (recursive ** is forbidden).
2. Lexicographic Order: Files are processed in alphabetical order.
3. Extension Inference:
   • .mp4, .mov, etc. infer the video subcommand.
   • .pdf infers extract-text.
   • Standard images default to analyze.
4. Sanitization: Input filenames are sanitized to a maximum of 64 characters using [A-Za-z0-9._-].

Sources: packages/scoutline/src/commands/vision.ts:430-482, packages/scoutline/src/commands/vision.ts:544-555

Prompt Substitution

The batch system supports prompt templates using {filename} and {filepath} tokens. The system replaces these tokens literally with the source file's base name or path before dispatching the request.

Sources: packages/scoutline/src/commands/vision.ts:494-500

Summary

Vision & Media Analysis in Scoutline enables complex investigative tasks through a robust multi-provider framework. By combining specialized operations like OCR and UI-to-code conversion with resilient media handling and batch processing, it provides a unified interface for programmatic visual research. Significant architectural features include the Z.AI URL-to-local fallback and the automated MiniMax operation mapping system.

Sources: packages/scoutline/src/commands/vision.ts:80-141, packages/scoutline/src/providers/minimax/vision-mappings.generated.ts:31-37