Touch & Apple Pencil Input
Relevant source files
The following files were used as context for generating this wiki page:
The Touch & Apple Pencil Input system manages high-fidelity user interactions across web and native Apple platforms. In the Apple native shell (macOS and iPadOS), the system leverages Metal-backed views to capture mouse, touch, and Apple Pencil events with high sampling frequency, supporting pressure, tilt, and hover capabilities. On the web platform, a Svelte-based state machine handles pointer events, including complex gestures like pinch-to-zoom and two-finger panning.
This system acts as a bridge between low-level hardware events and the project's core drawing logic, ensuring that coordinate transforms (screen-to-canvas) and tool applications are consistent regardless of the input device.
Native Input Architecture (Apple Shell)
The native Apple shell utilizes a specialized InputMTKView to handle hardware-specific events. This view subclasses MTKView and overrides standard event handling methods to capture granular data from the mouse on macOS and touches/Apple Pencil on iPadOS.
Component Breakdown
| Component | Responsibility |
|---|---|
InputMTKView | Subclass of MTKView that overrides mouseDown, mouseDragged, touchesBegan, and touchesMoved to detect input types (Mouse vs. Pencil) and report coordinates to a delegate. |
CanvasInputDelegate | A protocol defining callbacks for interaction starts, movements, and ends. |
PixelCanvasView | The SwiftUI view coordinator that implements CanvasInputDelegate, performing coordinate conversion and Bresenham interpolation before applying tools. |
EditorState | Holds the high-level interaction state, such as isDrawing, which is updated by the input handlers. |
Sources: apple/Dotorixel/Rendering/InputMTKView.swift, tasks/records/015-touch-pencil-input.md, apple/Dotorixel/State/EditorState.swift
Native Event Flow
The diagram below illustrates how a hardware event (like an Apple Pencil stroke) travels through the native rendering stack to update the canvas.
Web Interaction State Machine
On the web platform, interaction logic is decoupled from the DOM and encapsulated in a reactive state machine. This allows for rigorous testing of complex interaction states such as drawing, panning, and pinching.
Interaction Types
The state machine tracks the current mode of interaction to determine how pointer movements should be interpreted:
- Idle: No active interaction.
- Drawing: Single pointer (mouse or touch) is applying a tool to the canvas.
- Panning: User is moving the viewport (Space + Drag or Middle-click).
- Pinching: Multi-touch gesture for zooming and rotating/panning the viewport.
Sources: tasks/records/049-extract-canvas-interaction.md
Input Handling Logic
The CanvasInteraction module manages transitions between these states based on pointer counts and modifier keys:
Hardware Specifics & Implementation Details
Apple Pencil Support
The native shell specifically detects Apple Pencil input. InputMTKView uses UITouch properties on iPadOS to identify if a touch originates from a stylus. Future phases of the project aim to include hover previews and palm rejection for the Apple Pencil.
Sources: tasks/records/015-touch-pencil-input.md, issues/013-apple-native-catchup.md, CLAUDE.md
Coordinate Unification
To simplify cross-platform logic, the native shell sets isFlipped = true on macOS. This unifies the coordinate origin to the top-left across all platforms (web, macOS, and iPadOS), eliminating the need for Y-axis conversion branches in the coordinate math.
Sources: tasks/records/015-touch-pencil-input.md
Interaction Sampling & Redrawing
- Sampling: Native event handlers provide higher sampling frequency than standard SwiftUI
DragGesture, which is critical for smooth drawing. - Redrawing: Interaction state (e.g.,
lastPixel) is maintained in thePixelCanvasViewCoordinator rather thanEditorStateto avoid unnecessary SwiftUI view diffs during frame-rate inputs. Redraws are triggered by incrementing acanvasVersioncounter.
Sources: tasks/records/015-touch-pencil-input.md, apple/Dotorixel/Rendering/PixelCanvasView.swift
Technical Constraints Table
| Feature | Platform | Implementation Detail |
|---|---|---|
| Multi-touch | iPadOS | isMultipleTouchEnabled = false (Phase 1), deferred to specific zoom/pan tasks. |
| Mouse Cursor | macOS/Web | Move tool uses specific cursors; space-drag uses pan cursor. |
| Pressure/Tilt | Apple | Accessible via UITouch in InputMTKView. |
| Interpolation | All | Bresenham's algorithm ensures continuous lines between sampled points. |
Sources: tasks/records/015-touch-pencil-input.md, tasks/records/049-extract-canvas-interaction.md, issues/013-apple-native-catchup.md
Summary
The Touch & Apple Pencil Input system provides a robust, cross-platform foundation for pixel-perfect interaction. By utilizing platform-native hardware events on Apple devices and a formal state machine on the web, the system maintains high performance and accuracy. The architecture prioritizes coordinate unification and high-frequency sampling to ensure that the user experience remains fluid, whether using a mouse, a finger, or an Apple Pencil.