feat: Make screenshot mouse coordinates easier to verify

Merged
HA
3mo ago
+1388-114

Summary

  • Mouse-related tools now report screenshot, input, and injected Unity coordinate details so agents can verify clicks without guessing Y-axis conversions.
  • Rendering screenshots can annotate UI and 3D raycast candidate points, and uloop raycast can check what a Game View coordinate hits before clicking.

User Impact

  • Before, screenshot pixels and Mouse.current.position could look contradictory when the Game View render texture did not match the Unity input area.
  • After, responses show coordinate systems, Game View size, injected Unity position, and screenshot-to-input formulas, including render-image Y offsets.
  • 3D click targets can be checked from screenshots before sending mouse input.

Changes

  • Added shared Game View coordinate conversion metadata to screenshot, simulate-mouse-input, simulate-mouse-ui, and raycast tool docs and schemas.
  • Added a raycast command and raycast-grid screenshot annotations that account for render texture offsets and synced physics transforms.
  • Declared the Unity 3D Physics module dependency and added focused EditMode and PlayMode coverage.

Verification

  • uloop compile
  • uloop run-tests --test-mode EditMode --filter-type regex --filter-value "(GameViewCoordinateUtilityTests|RaycastGridAnnotatorTests|RaycastToolTests|EditorWindowCaptureUtilityTests)"
  • uloop run-tests --test-mode PlayMode --filter-type exact --filter-value "Tests.PlayMode.SimulateMouseInputTests"
  • npm run build
  • autoreview --mode branch --base origin/main

Summary

  • Added a new raycast MCP tool (schema + response DTO + docs) that raycasts from Camera.main using top-left Game View (X,Y) coordinates and returns hit details plus shared coordinate-conversion metadata.
  • Unified the mouse/screenshot coordinate contract across screenshot, simulate-mouse-input, and simulate-mouse-ui by standardizing coordinate system naming (top-left game view vs bottom-left injected Unity), Game View dimensions, screenshot→input conversion formulas, and injected Unity positions (including the internal Y-axis handling rules).
  • Implemented rendering-aware screenshot metadata by refactoring capture/coordinate mapping to account for render-texture height vs Game View size (ImageToInputOffsetY / screenshot-to-input offset), and updated screenshot response/contracts to use ImageCoordinateSystem and conversion/offset fields.
  • Added raycast-grid annotation support to screenshot (AnnotateRaycastGrid) and introduced shared point/overlay models (RaycastGridPointInfo, grid annotator that syncs physics and produces injected Unity positions + hit labels).
  • Added editor utilities for coordinate conversion and physics raycasting in Game View space (GameViewCoordinateUtility, GameViewRaycastUtility) and physics-transform sync behavior needed for reliable raycast results.
  • Declared the Unity 3D Physics module dependency and updated bundled CLI/tool documentation and tool reference text to reflect the new coordinate contract and the added uloop raycast flow.
  • Expanded test coverage in EditMode and PlayMode to validate coordinate conversion, render-texture top offset computation, raycast-grid sampling/overlay filtering, injected coordinate metadata for simulated mouse input, and raycast hit/miss/error cases (including camera-missing and Physics.autoSyncTransforms handling).