feat: Screenshot annotation lists UI Toolkit controls, and simulate-mouse-ui clicks and drags them

.agents/skills/uloop-screenshot/SKILL.md

@@ -23,7 +23,7 @@ uloop screenshot [--window-name <name>] [--resolution-scale <scale>] [--match-mo
 | `--match-mode` | enum | `exact` | Window name matching mode: `exact`, `prefix`, or `contains`. Ignored when the resolved capture mode is rendering (auto during Play Mode resolves to rendering). |
 | `--capture-mode` | enum | `auto` | `auto` - rendering in PlayMode and window otherwise, `window` - capture EditorWindow including toolbar, `rendering` - capture game rendering only (PlayMode required), `GameView` - alias for `rendering`. Rendering screenshots return `ScreenshotToInputFormula` for converting raw image pixels before calling simulate-mouse-input (including `--dry-run`) or simulate-mouse-ui. |
 | `--output-directory` | string | `""` | Output directory path for saving screenshots. When empty, uses default path (.uloop/outputs/Screenshots/). Accepts absolute paths. |
-| `--annotate-elements` | flag | - | Annotate interactive UI elements with index labels and interaction hints (A / CLICK, B / DRAG, ...). The response includes an `AnnotatedElements` array with element metadata sorted by `SortingOrder` then `SiblingIndex` (an approximation of z-order, not the guaranteed draw order). Only works when the resolved capture mode is rendering (auto during Play Mode resolves to rendering) in PlayMode. |
+| `--annotate-elements` | flag | - | Annotate interactive uGUI elements and UI Toolkit controls on screen-space panels with index labels and interaction hints (A / CLICK, B / DRAG, ...). The response includes an `AnnotatedElements` array with element metadata sorted by `SortingOrder` then `SiblingIndex` (an approximation of z-order, not the guaranteed draw order). Only works when the resolved capture mode is rendering (auto during Play Mode resolves to rendering) in PlayMode. |
 | `--annotate-raycast-grid` | flag | - | Annotate clustered 3D physics collider candidates as `PhysicsCollider` entries in `AnnotatedElements`. Uses `Camera.main` visibility and the same top-left Game View coordinates as `simulate-mouse-input`. Only works when the resolved capture mode is rendering (auto during Play Mode resolves to rendering) in PlayMode. |
 | `--raycast-layer-mask` | string | `""` | Comma-separated physics layer names to narrow which layers `--annotate-raycast-grid` clusters. Hits are limited to layers also visible to `Camera.main.cullingMask`. When omitted, clusters against `Physics.DefaultRaycastLayers`. |
 | `--elements-only` | flag | - | Return only annotated element JSON without capturing a screenshot image. Requires `--annotate-elements` or `--annotate-raycast-grid`, and the resolved capture mode to be rendering (auto during Play Mode resolves to rendering) in PlayMode. |

.agents/skills/uloop-screenshot/references/annotated-elements.md

@@ -10,15 +10,16 @@ Device Simulator is supported for this flow: prefer `--capture-mode rendering` (
 
 - `Label`: Index label in JSON (`A` = first in that sort, `B` = next, ...). Treat it as a stable handle, not as proof of what is actually in front: when overlapping elements cross parents, pick the target by `Path` and confirm the result with `simulate-mouse-ui`'s `HitGameObjectName` — the handler that received the events, resolved up the hierarchy from the raycast hit (or from `--target-path` in bypass mode), so it can name an ancestor of the element you clicked. Screenshot labels also include the interaction hint, such as `A / CLICK` or `B / DRAG`.
 - `Name`: Element name
-- `Path`: Hierarchy path from the scene root, for example `Canvas/Panel/Button`. Use this as `simulate-mouse-ui --target-path` when bypassing raycast blockers.
+- `Path`: Hierarchy path from the scene root, for example `Canvas/Panel/Button`. Use this as `simulate-mouse-ui --target-path` when bypassing raycast blockers. UI Toolkit entries use a different form that `--target-path` does not accept; click them at `SimX`/`SimY` instead. See [UI Toolkit Elements](#ui-toolkit-elements).
 - `Type`: Element type (`Button`, `Toggle`, `Slider`, `Dropdown`, `InputField`, `Scrollbar`, `Draggable`, `DropTarget`, `Selectable`, `PhysicsCollider`)
 - `Interaction`: Derived interaction category (`Click`, `Drag`, `Drop`, `Text`) or `Raycast` for clustered physics collider entries. Use this to choose between `simulate-mouse-ui --action Click`, drag actions, or `simulate-mouse-input` (including `--dry-run`).
+- `UiSystem`: UI system the entry came from: `UGUI`, `UIToolkit`, or empty for `PhysicsCollider` entries.
 - `Layer`: Physics layer name for `PhysicsCollider` entries. Empty for UI entries.
 - `Components`: Collider and MonoBehaviour component type names from the hit GameObject for `PhysicsCollider` entries. Empty for UI entries.
 - `SimX`, `SimY`: Target click position in top-left Game View coordinates. For UI entries this is the element center when that center is raycast-reachable; otherwise it is a reachable probe point inside the bounds (center, then four interior quarter points). For `PhysicsCollider` entries this is a representative sampled hit. Use these directly with `simulate-mouse-ui --x/--y` or `simulate-mouse-input --x/--y` (`--dry-run` optional).
 - `BoundsMinX`, `BoundsMinY`, `BoundsMaxX`, `BoundsMaxY`: Bounding box in the same coordinates as `SimX/SimY`. For `PhysicsCollider` entries, this is the axis-aligned sampled-cell coverage box from reachable raycast hits, not a guarantee that every interior point is clickable.
-- `SortingOrder`: Canvas sorting order. Higher values are in front.
-- `SiblingIndex`: Transform sibling index under the element's direct parent. Do not use it as a reliable z-order signal across nested UI hierarchies; the frontmost-first sort above can then disagree with draw order.
+- `SortingOrder`: Canvas sorting order (for UI Toolkit, the panel's `PanelSettings.sortingOrder` rounded down). Higher values are in front.
+- `SiblingIndex`: Transform sibling index under the element's direct parent (for UI Toolkit, the index in the parent visual element). Do not use it as a reliable z-order signal across nested UI hierarchies; the frontmost-first sort above can then disagree with draw order.
 
 ## Exclusion rules (by design)
 
@@ -29,6 +30,33 @@ The annotator lists elements that share the same EventSystem raycast path as `si
 - Elements whose center and four interior quarter probe points are all covered by another raycast hit are not listed.
 - Selectables that are not interactable (`interactable = false`, or a parent `CanvasGroup` with `interactable = false`) ignore clicks and drags, so they are not listed. A separate pointer handler script on the same GameObject is still listed as that handler.
 
+## UI Toolkit Elements
+
+Entries with `UiSystem` `UIToolkit` come from the `UIDocument`s on screen-space panels shown in the Game View. They use the same fields and top-left Game View coordinates as uGUI entries.
+
+- `Type`: `Button` (`Button`, `RepeatButton`), `Toggle` (`Toggle` and other boolean fields), `Slider` (`Slider`, `SliderInt`, `MinMaxSlider`), `Scrollbar` (`Scroller`, including the scrollers of a `ScrollView`), `InputField` (`TextField`), `Dropdown` (`DropdownField`, `EnumField`). `Interaction` follows from `Type` as for uGUI.
+- `Path`: `<UIDocument GameObject path>#<element chain>`, for example `UI/Menu#panel/play`. The chain starts below the document root and joins element names with `/`; an element without a name appears as its type name.
+
+Omitted on purpose:
+
+- Disabled elements, `visibility: hidden`, `display: none` (with everything under it), and zero-size elements.
+- Elements covered at the center and every interior quarter probe point by a panel with a higher sorting order or by another element of the same panel, including parts clipped away by a parent with `overflow: hidden`.
+- Panels that render into a RenderTexture, panels on a display other than the Game View's, world-space panels (Unity 6), and panels whose screen-to-panel mapping is not a plain scale and offset.
+- Parts inside a control (the slider and buttons of a `Scroller`, for example) and any element that is not one of the types above. A custom element with only a `Clickable` manipulator is not listed.
+
+Limits:
+
+- When uGUI and UI Toolkit overlap, which one is in front is not checked; both may be listed.
+- Panels with the same sorting order are not checked against each other.
+- `Bounds` is the whole element rectangle and can include parts clipped away by a parent.
+- Unity 2022.3 with both Device Simulator and Game View open lays UI Toolkit panels out at the Game View size and stretches them onto the simulated screen. Bounds and `SimX`/`SimY` then match where clicks land but not the PNG, and `set-game-view-size` cannot change the hidden Game View while Simulator is shown. Closing the Game View while using Simulator makes them match.
+
+Clicking:
+
+- `simulate-mouse-ui --action Click --x <SimX> --y <SimY>` (or `LongPress`, or a drag from that point) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values.
+- `simulate-mouse-input --action Click --x <SimX> --y <SimY>` also reaches them when the scene has an EventSystem with an Input System UI input module.
+- Without an EventSystem the `simulate-mouse-input` click does not reach UI Toolkit controls.
+
 ### PhysicsCollider Entries Per Closed Region
 
 When a single `PhysicsCollider` GameObject's reachable raycast samples form multiple 4-connected regions on screen — typically because UI occlusion splits them apart — each region becomes its own `AnnotatedElements` entry. `Path`, `Name`, `Layer`, and `Components` are identical across those entries (they describe the same GameObject), while `Label`, `Bounds`, `SimX`, `SimY`, and `RaycastOutlineSegments` are independent per region so each closed area is separately addressable.

.agents/skills/uloop-simulate-mouse-input/SKILL.md

@@ -67,7 +67,7 @@ All rows below assume the New Input System is installed.
 
 | Scenario | Tool |
 |----------|------|
-| Click a Unity UI Button (IPointerClickHandler) | `simulate-mouse-ui` |
+| Click a uGUI or UI Toolkit button | `simulate-mouse-ui` |
 | Runtime logic reads `Mouse.current.leftButton` | `simulate-mouse-input` |
 | Runtime logic reads right-click | `simulate-mouse-input --button Right` |
 | Drag a UI slider | `simulate-mouse-ui --action Drag` |
@@ -110,7 +110,7 @@ uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --dura
 ## Coordinate System
 
 `--x`/`--y` use top-left Game View coordinates: pass `AnnotatedElements[].SimX/SimY`
-directly, convert raw screenshot pixels with `ScreenshotToInputFormula`, and never flip Y
+directly (UI Toolkit entries need an EventSystem with an Input System UI module here; `simulate-mouse-ui` reaches them without one), convert raw screenshot pixels with `ScreenshotToInputFormula`, and never flip Y
 in the caller (the tool converts internally). Conversion formula, Device Simulator
 notes, and how the injected position reads back inside Unity are in
 `references/output-and-coordinates.md`.

.agents/skills/uloop-simulate-mouse-input/references/output-and-coordinates.md

@@ -30,7 +30,7 @@ Verify visual outcome with a follow-up screenshot.
 
 - `--x` / `--y` use **top-left Game View coordinates**.
 - Raw image pixels from `uloop screenshot --capture-mode rendering` must be converted with `ScreenshotToInputFormula`.
-- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool.
+- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. For UI Toolkit entries (`UiSystem` `UIToolkit`), a `Click` reaches the control only when the scene has an EventSystem with an Input System UI input module; without one, nothing receives the click. `simulate-mouse-ui` reaches them without an EventSystem.
 - Do not flip Y in the caller. The tool converts internally for Unity Input System:
 
 ```text

.agents/skills/uloop-simulate-mouse-ui/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: uloop-simulate-mouse-ui
 toolName: simulate-mouse-ui
-description: "Simulate PlayMode EventSystem UI mouse actions using screen coordinates. Use for UI clicks, long-presses, or drags from annotated screenshots."
+description: "Simulate PlayMode UI mouse actions on uGUI and UI Toolkit using screen coordinates. Use for UI clicks, long-presses, or drags from annotated screenshots."
 ---
 
 # Task
@@ -64,6 +64,7 @@ uloop simulate-mouse-ui --action <action> --x <x> --y <y> [options]
 - Get coordinates from `AnnotatedElements` JSON (`SimX`/`SimY`) — do NOT look up GameObject positions
 - `--bypass-raycast` still uses coordinates for pointer event positions, but chooses the clicked, long-pressed, or dragged GameObject by `--target-path`
 - If `--target-path` or `--drop-target-path` matches multiple active GameObjects, the command fails instead of choosing an arbitrary duplicate
+- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked, long-pressed, and dragged at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`.
 - Device Simulator play view is supported. Prefer `uloop screenshot --capture-mode rendering --annotate-elements` for coordinates; they use the simulated device resolution (`Handles.GetMainGameViewSize()` / `Screen`), not the Simulator chrome scale.
 
 ## Pause Point Inspection (Standard for E2E)
@@ -85,12 +86,9 @@ uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300
 # Long-press a button for 3 seconds
 uloop simulate-mouse-ui --action LongPress --x 400 --y 300 --duration 3.0
 
-# One-shot drag (start to end in one call)
+# One-shot drag (add --drag-speed 200 to slow it for visual inspection)
 uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300
 
-# Slow drag for visual inspection
-uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 --drag-speed 200
-
 # Split drag with hold (for inspection between steps)
 uloop simulate-mouse-ui --action DragStart --x 400 --y 300
 uloop screenshot --window-name Game
@@ -101,14 +99,14 @@ uloop simulate-mouse-ui --action DragEnd --x 600 --y 300
 ## Prerequisites
 
 - Unity must be in **PlayMode**
-- Target scene must have an **EventSystem** GameObject
-- UI elements must have a **GraphicRaycaster** on their Canvas
+- uGUI targets need an **EventSystem** and a **GraphicRaycaster** on their Canvas; UI Toolkit targets need neither
 - If you need runtime mouse input rather than UI pointer events, `simulate-mouse-input` assumes the project uses the New Input System; otherwise prefer `execute-dynamic-code`
 
 ## Output
 
 The response reports `Success`, `Message`, `Action`, `HitGameObjectName`, the used
-coordinates, and `InterruptedByPausePoint`/`PausePointHits`. Two readings that matter:
+coordinates, and `InterruptedByPausePoint`/`PausePointHits`; `UiSystem` and `HitPath` say
+which UI system was hit and give its path in the annotation's form. Two readings that matter:
 `Click`/`LongPress` on empty space still return `Success = true` with
 `HitGameObjectName = null` (drag actions on empty space fail), and on a pause-point
 interruption `Message` states whether the pointer event fired before the pause.

.agents/skills/uloop-simulate-mouse-ui/references/output.md

@@ -5,7 +5,9 @@ Returns JSON with:
 - `Success`: Whether the operation succeeded
 - `Message`: Status message (e.g. "Hit element: ButtonStart" or "No UI element under (x, y)")
 - `Action`: Echoes which action was executed (`Click`, `Drag`, `DragStart`, `DragMove`, `DragEnd`, or `LongPress`)
-- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself
+- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the name of the outermost control an annotation would list (the same as the annotation's `Name`); when the picked element is not inside such a control (a bare `Label` or `VisualElement`), it is that element's own `name`, or its type name when empty, and the element is not annotated. Unlike uGUI, which reports null when no handler is found, UI Toolkit reports the picked element whether or not it handles pointer events
+- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` and drags, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object
+- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` and drags): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls. Null when nothing was hit, or when the hit element belongs to no document, such as the menu of an open dropdown
 - `PositionX`: Target X coordinate that was used
 - `PositionY`: Target Y coordinate that was used
 - `EndPositionX`: Drag end X coordinate (nullable float; populated for drag actions only)
@@ -15,3 +17,7 @@ Returns JSON with:
 Verify the visual outcome with a follow-up `uloop screenshot --capture-mode rendering --annotate-elements`.
 
 Note: Click and LongPress on empty space (no UI element) still return `Success = true` with `HitGameObjectName = null`. Drag actions on empty space return `Success = false`.
+
+## UI Toolkit Elements
+
+`Click`, `LongPress`, and the drag actions reach UI Toolkit controls at their annotated `SimX`/`SimY` (`--from-x`/`--from-y` for `Drag`). How presses and drags are delivered and what is not supported: `ui-toolkit.md`.