@smbcloud / cli
Code Commits Issues Pulls Cloud Agent CI/CD
release/v0.7.4
claude/smbcloud-github-app-k9641p development feature/apps feature/build-release feature/ci-docs feature/ci-ruby-gems feature/cicd feature/completion feature/dist-nuget feature/fix-ci-sdk-pypi-rubygem feature/fix-clippy-on-stable-channel feature/fix-deployment-script feature/github-email feature/gresiq-v1 feature/project-new-structure feature/rsync-deploy feature/ruby-runner feature/sdk-npm feature/sdk-workflows feature/smbcloud-cli-release-npm feature/smbcloud-deploy feature/wasm main release/v0.7.4
md 262 lines 10.2 KB
Rendered Raw
1 # ControlKit runners
2
3 ControlKit is the native XCTest runner used by the CLI's Apple-platform MCP
4 tools. The public runner project is
5 [`xcrs-controlkit`](https://github.com/ondeinference/xcrs-controlkit). It uses
6 one Xcode project generated by XcodeGen, with platform-specific targets and
7 schemes:
8
9 | Platform | Scheme | App product |
10 | --- | --- | --- |
11 | iOS / iPadOS | `ControlKit-iOS` | `ControlKit.app` |
12 | tvOS | `ControlKit-tvOS` | `ControlKit.app` |
13 | visionOS | `ControlKit-visionOS` | `ControlKit.app` |
14 | watchOS | `ControlKit-watchOS` | `ControlKit.app` |
15 | macOS | `ControlKit-macOS` | `ControlKit.app` |
16
17 The scheme and target names keep their platform suffixes so Xcode can select
18 the correct SDK, while each built app is named `ControlKit`.
19
20 ## Build a runner
21
22 Install XcodeGen once, clone the runner project, and generate the Xcode
23 project:
24
25 ```sh
26 brew install xcodegen
27 cd <path-to-xcrs-controlkit>/Runners
28 xcodegen generate
29 ```
30
31 Build the simulator runner for the platform you need:
32
33 ```sh
34 xcodebuild -project XCRSControlKitRunner.xcodeproj \
35 -scheme ControlKit-<platform> \
36 -destination 'generic/platform=<platform> Simulator' \
37 build-for-testing
38 ```
39
40 Replace `<platform>` with `iOS`, `tvOS`, `watchOS`, or `visionOS`. macOS is
41 built with `-destination 'platform=macOS'` and runs on the local Mac rather
42 than a simulator.
43
44 ## Start the runner
45
46 Boot a compatible simulator and run the UI-test host:
47
48 ```sh
49 xcodebuild -project XCRSControlKitRunner.xcodeproj \
50 -scheme ControlKit-<platform> \
51 -destination 'id=<simulator-udid>' \
52 test-without-building
53 ```
54
55 The shared XCTest runner starts a local ControlKit JSON-RPC server on
56 `127.0.0.1:12004`. Keep that test process running while calling the MCP tools.
57 The macOS UI-test target includes the network-server entitlement required to
58 bind the local port.
59
60 ## Run on a physical device
61
62 The same runner also works installed on a real iOS, tvOS, watchOS, or
63 visionOS device, not just a simulator. Build and start it against the
64 device's UDID (find it with `xcrun devicectl list devices`) instead of a
65 simulator destination:
66
67 ```sh
68 xcodebuild -project XCRSControlKitRunner.xcodeproj \
69 -scheme ControlKit-<platform> \
70 -destination 'id=<device-udid>' \
71 build-for-testing
72
73 TEST_RUNNER_CONTROLKIT_LISTEN_HOST=0.0.0.0 xcodebuild \
74 -project XCRSControlKitRunner.xcodeproj \
75 -scheme ControlKit-<platform> \
76 -destination 'id=<device-udid>' \
77 test-without-building
78 ```
79
80 The `CONTROLKIT_LISTEN_HOST` environment variable controls what the
81 in-process RPC server binds to (`Shared/ControlKitRPCServer.swift`) — it
82 defaults to `127.0.0.1`, which is loopback-only *on the device itself* and
83 unreachable from your Mac. `xcodebuild` forwards any `TEST_RUNNER_`-prefixed
84 environment variable into the test process with the prefix stripped, so
85 setting `TEST_RUNNER_CONTROLKIT_LISTEN_HOST=0.0.0.0` makes the server bind on
86 every interface, including the one reachable over your LAN. Find the device's
87 address with its Bonjour hostname (`<device-name>.local`, discoverable via
88 `dns-sd -B _airplay._tcp local` for an Apple TV) or its CoreDevice tunnel
89 address (`xcrun devicectl device info details --device <udid>`, under
90 `Tunnel IP Address`).
91
92 ## Connect through the automation MCP profile
93
94 Start either automation server:
95
96 ```sh
97 xcrs --mcp
98 # or
99 smb --mcp --scope automation
100 ```
101
102 The runner tools accept either `simulator_name` or `simulator_udid` for a
103 simulator, or `host` (plus `controlkit_port`, defaulting to `12004`) for a
104 physical device or any other remote runner. Call `device_select` once to
105 remember the target so later tools need no target arguments. Both commands
106 expose the same unprefixed tool names.
107
108 | Tool | Runner | Purpose |
109 | --- | --- | --- |
110 | `device_select` | All platforms | Remember the active simulator or device for later tools. |
111 | `device_capabilities` | All platforms | Read platform and capability information. |
112 | `input_click` | macOS | Click at screen coordinates. |
113 | `input_spatial_tap` | visionOS | Perform a spatial tap. |
114 | `input_tap` | iOS / tvOS / watchOS, simulator or physical | Perform a touch tap. |
115 | `input_text` | iOS / tvOS, simulator or physical | Type into the focused field. |
116 | `input_swipe` | iOS / tvOS, simulator or physical | Swipe between coordinates. |
117 | `input_button` | iOS / tvOS, simulator or physical | Press a Home or tvOS remote button. |
118 | `app_launch`/`app_terminate` | simulator or physical | Launch/terminate an app by bundle ID. On a physical device this calls the runner's `device.apps.launch`/`device.apps.terminate` RPC methods instead of `simctl`. |
119 | `screen_capture` | Simulator or physical | Capture a PNG screenshot. Uses `simctl` for a simulator, or `devicectl device capture screenshot` when a `device` identifier is given. |
120 | `ui_describe` | All UI-test runners | Read the accessibility hierarchy for the supplied `bundle_id`. |
121 | `ui_element_list` | All UI-test runners | Read actionable elements with tap coordinates for the supplied `bundle_id`. |
122 | `ui_tap` | All UI-test runners | Resolve and activate an accessibility element by label, identifier, or value. |
123
124 UI introspection requires a recent ControlKit runner that implements
125 `device.dump.ui`. The `ui_tap` action requires a runner that implements
126 `device.ui.tap`; it resolves and activates the requested element inside the
127 target app, so an MCP client does not need to turn accessibility coordinates
128 into a second action. The label, identifier, or value must match exactly one
129 element. On tvOS, that element must already have focus; use `input_button` to
130 navigate focus before calling `ui_tap`. All three UI tools require the target
131 app's `bundle_id`; this lets XCTest attach to an already-running app instead of
132 accidentally reading the ControlKit host app.
133
134 For a local macOS runner, omit simulator fields and pass `host` and
135 `controlkit_port` when needed:
136
137 ```json
138 {
139 "host": "127.0.0.1",
140 "controlkit_port": 12004,
141 "x": 400,
142 "y": 300
143 }
144 ```
145
146 For a physical device, pass its Bonjour hostname or tunnel address as `host`
147 instead:
148
149 ```json
150 {
151 "host": "<device-name>.local",
152 "controlkit_port": 12004,
153 "button": "select"
154 }
155 ```
156
157 Platform-specific operations intentionally reject unsupported input. For
158 example, visionOS rejects raw touch, watchOS rejects pointer and spatial input,
159 and macOS rejects touch and Home-button requests.
160
161 ## Inspect the macOS host
162
163 The `host` subcommands describe what is actually on the local Mac's screen.
164 They are for the cases where a UI test's own output is not enough: which
165 windows are up, what text a screenshot contains, or which colors run along one
166 line of pixels. Unlike the runner tools above they talk to no device and start
167 no XCTest — they run a small Swift helper,
168 [`xcrs-host-inspect`](https://github.com/ondeinference/xcrs-controlkit), in
169 process on the host. They need macOS 13 or later for the AppKit window server
170 and Vision's text recognizer, are CLI-only, and are not part of the `--mcp`
171 tool set.
172
173 ### Build the helper
174
175 Build it once with SwiftPM from the `xcrs-controlkit` checkout:
176
177 ```sh
178 cd <path-to-xcrs-controlkit>
179 swift build -c release
180 ```
181
182 The CLI uses the first `xcrs-host-inspect` it finds, in this order:
183
184 1. the path in the `XCRS_HOST_INSPECT` environment variable,
185 2. next to the `xcrs` binary itself, for a release that ships the helper,
186 3. `.build/release`, then `.build/debug`, relative to the current directory,
187 4. `~/bin`, `/usr/local/bin`, and `/opt/homebrew/bin`.
188
189 Because steps 3 are relative, running `xcrs` from inside the checkout finds a
190 `swift build` without any further setup. Set `XCRS_HOST_INSPECT` when you have
191 several builds and want to choose between them.
192
193 ### Read the screen
194
195 List the host's on-screen windows, ordered by layer and then window id:
196
197 ```sh
198 xcrs host windows
199 xcrs host windows --owner Mail --json
200 ```
201
202 Each window reports its `id`, the owning application name (`owner`), its
203 `title`, its `layer`, and its `bounds`. `--owner` keeps only windows owned by
204 one application name.
205
206 Read the text in an image with Vision and print each string with its
207 confidence:
208
209 ```sh
210 xcrs host ocr --image shot.png
211 xcrs host ocr --image shot.png --language en-US --language sv-SE --json
212 ```
213
214 `--language` is repeatable; omit it to use the platform's own default.
215
216 Crop a rect out of an image and write it to a new PNG:
217
218 ```sh
219 xcrs host crop --image shot.png --rect 20,40,320,180 --out header.png
220 ```
221
222 `--rect` takes `x,y,width,height`. The result reports the written path and its
223 pixel dimensions, and the input image is never modified.
224
225 Probe one row or column and describe it as a run of colors, so a solid surface
226 collapses to a single line instead of thousands of pixels:
227
228 ```sh
229 xcrs host probe --image shot.png --row --position 300
230 xcrs host probe --image shot.png --column --position 120 --from 40 --to 260
231 ```
232
233 `--position` is the row or column to read, and exactly one of `--row` or
234 `--column` is required. `--from` and `--to` clip the line to part of its
235 length. Runs report `start`, `end`, `width`, and a quantized `#rrggbb`
236 `color`, so low-order noise does not split one surface into several runs.
237
238 ### Coordinates and scale
239
240 Coordinates you pass are in caller units, and `--scale` says how many pixels
241 one unit covers: the default `1` reads them as pixels, and `2` reads them as
242 Retina points on a 2x display. `crop` and `probe` honor it for their rect,
243 position, and bounds. Output geometry follows the same convention — an image
244 reports `width`/`height` in caller units alongside `pixel_width`/
245 `pixel_height` — while window bounds are screen points regardless of
246 `--scale`.
247
248 ### Exit codes and errors
249
250 The helper separates a malformed command from failed work: it exits with
251 status 2 when it rejects the arguments, and 1 when the operation itself fails,
252 writing the reason to stderr. The CLI validates rects, the scale, and probe
253 bounds before it spawns the helper, so bad input usually never reaches it.
254 When the helper does fail, the CLI reports it with its status — for example
255 `xcrs-host-inspect crop exited with status 1: …` — and exits non-zero. A
256 helper that cannot be started at all, such as a missing binary, is reported
257 with a hint to run `swift build` in `xcrs-controlkit`.
258
259 ## See also
260
261 For the product-level overview of end-to-end testing with xcrs and
262 ControlKit, see [smbcloud.xyz/xcrs-end-to-end-testing](https://smbcloud.xyz/xcrs-end-to-end-testing).