PR #30940 - manaflow-ai/cmux

What

When an agent looks at a video's frames through the cmux video_frames agent tool, the transcript now draws a compact video card instead of the raw call:

The row says what it did: Looked at 8 video frames.
Under it, starting at the row's text column, the video's name and length (and size), then a filmstrip of the frames, each with its time in a corner (0:02, or 0:02.5 when whole seconds would repeat).
Click a frame to open it large in the pane's existing image viewer.
Click the name to play the video in Quick Look. This is the same viewer an attached video's tile opens (video.quickLook), and the host checks the path the same way. In a read-only (guest) transcript the name is plain text.
Opened, the call shows the tool's note (length, size, frame times) without the frames' base64.

Every frame box takes the video's shape before it decodes, so the transcript never shifts. With many frames the strip scrolls sideways at the row's width instead of growing taller.

Leo, 2026-11-01, looking at two screenshots of an agent's look at an attached video (a Shell card with a long ffmpeg command and the full Chat Uploads path printed twice, then Read /tmp/vidframes.../grid.png with a contact sheet): "i feel like cmux-next should give u better tools to deal with this kind of thing ... you can see what we can work on", then "can you help work on all these".

Source shape

The video_frames tool is #30921 (open). videoFrames.ts reads its exact output, and the gallery and a test use that wording. The output is one image/jpeg block per frame and one text block. The text's header line is <path>: 16.40 s, 1920x1080, 30 fps, no audio. 8 frames below, ..., which gives the resolved path, the length and the size. Each frame then has a #N at T s line with its time. Frames may arrive as ACP image blocks (tool.images) or inside an MCP envelope in the output, and are never deduped, so a still screen's identical frames keep their times. Other wordings still parse: key: value notes, numbered lists and JSON. Detection uses the MCP tool name (mcp__<server>__video_frames, or Codex's {server, tool}). If a call's frames can't be found, it draws exactly as before.

It's parsed once per immutable tool snapshot (a WeakMap), never per render or per streamed token. The card adds no timer, observer or animation loop. Frame images use loading="lazy".

Not changed

Ordinary Shell cards and every other tool row render exactly as before. Only McpCallRow changes, and only for a video_frames call.
The sent user bubble for an attached video: a separate open lane already changes that bubble and the video's prompt (videoPrompt.ts). Its stated design is that the bubble draws the video's own tile and never the frames, so this PR leaves the bubble alone.

Gallery

agent-pane.video-frames (VideoFramesCard.gallery.tsx), built from the real ACP updates through mergeToolItem and drawn by the real McpCallRows:

Variant	What
`strip` (chosen)	name and length above a one-row filmstrip
`grid`	the frames as a contact sheet that wraps
`poster`	the first frame large, the rest in a strip under it, the name below
`strip-many`	16 frames: the strip scrolls sideways
`strip-unnamed`	no file named: length and frames, no name to play

Why strip: it's what was asked for (a filmstrip). It's the most compact, under 90px tall at any frame count, so a look at a video takes about as much room as one image strip and never pushes the turn's reply off screen. It also reads in time order left to right. grid gets tall fast at 8 to 16 frames, and poster repeats the first frame at a size the image viewer already gives on click.

Design spec: design/chat-transcript.md (leo-workspace), tool rows. No settled design is replaced.

Tests

videoFrames.test.ts: Claude and Codex call shapes, an MCP envelope with a JSON note, other note wordings and paths with spaces, untimed frames, and non-calls (other tools, running, failed, frameless, a shell ffmpeg).
Fleet run (cmux-ci test): the unit tests, gallery coverage, typecheck, design tokens, catalog keys, fmt and lint on the changed files.
Two new strings in every shipped language: tools.videoFrames (a plural) and media.videoFrameAt. The play button reuses composer.playVideo.

Follow-ups

Two video_frames calls in a row group under the generic MCP group line, and their cards show only once the group is opened. A call made inside a cmux_exec script gets no card.
The name shows as playable even for a path the host refuses (e.g. /tmp). The refusal is silent.
Let the image viewer step through a card's frames with the arrow keys (today it opens one frame).

🤖 Generated with Claude Code

Agent session: agent session