@smbcloud / cli / commits / 5896090

Add host inspection commands to xcrs (#78)

* Add host inspection commands to xcrs Wrap the `xcrs-host-inspect` Swift helper so a control run can read the macOS host directly: `host windows` lists on-screen windows, `host ocr` recognizes image text, `host crop` writes a cropped PNG, and `host probe` reads one line of an image as runs of color. Every command takes `--scale`, the pixels per caller unit, so a Retina screenshot can be addressed in points while the helper still reports pixel geometry beside it. The client treats a non-zero helper exit as data rather than an error, so the helper's own status and message reach the caller instead of being flattened. Requests are validated before the helper is spawned, and only a helper that cannot be started at all surfaces as a launch failure. Co-Authored-By: siGit Code v1.7.0-acp <noreply@sigit.si> * Document the xcrs host inspection commands Cover the four `xcrs host` subcommands in docs/controlkit.md: build the Swift helper, how the CLI discovers it (XCRS_HOST_INSPECT and the search order), each command's flags, the --scale caller-unit convention, and how the helper's exit codes surface. Co-Authored-By: siGit Code v1.7.0-acp <noreply@sigit.si> --------- Co-authored-by: siGit Code v1.7.0-acp <noreply@sigit.si>

keypair34 committed Oct 17, 2026 at 13:58 UTC 589609051727e5eb43c6db778c28e32a81b52a8d
3 files changed +1434 -1
crates/xcrs/src/main.rs
+352 -1
@@ -1,7 +1,11 @@
1 use anyhow::{anyhow, Result};
2 use clap::{Parser, Subcommand};
3 use std::path::PathBuf;
4 -use xcrs::{AndroidDebugBridge, AndroidKit, IosAppTest, XcodeCommandLineTools};
4 +use xcrs::{
5 + AndroidDebugBridge, AndroidKit, HostCropRequest, HostInspect, HostInspectRect,
6 + HostInspectScale, HostOcrRequest, HostProbeAxis, HostProbeRequest, IosAppTest,
7 + XcodeCommandLineTools,
8 +};
9
10 #[derive(Debug, Parser)]
11 #[command(name = "xcrs")]
@@ -21,6 +25,9 @@ enum Commands {
25 Simulator(SimulatorCommand),
26 #[command(subcommand)]
27 AndroidKit(AndroidKitCommand),
28 + /// Inspect the macOS host: on-screen windows, image text, crops, and pixel probes.
29 + #[command(subcommand)]
30 + Host(HostCommand),
31 IosAppTest {
32 #[arg(long, conflicts_with = "simulator_udid")]
33 simulator_name: Option<String>,
@@ -61,6 +68,88 @@ enum AndroidKitCommand {
68 },
69 }
70
71 +#[derive(Debug, Subcommand)]
72 +enum HostCommand {
73 + /// List the host's on-screen windows, ordered by layer and then id.
74 + Windows {
75 + /// Only windows owned by this application name.
76 + #[arg(long)]
77 + owner: Option<String>,
78 + #[arg(long)]
79 + json: bool,
80 + },
81 + /// Read the text in an image with the platform's Vision text recognizer.
82 + Ocr {
83 + #[arg(long)]
84 + image: PathBuf,
85 + /// Recognition language, repeatable; omit for the platform's own default.
86 + #[arg(long = "language")]
87 + languages: Vec<String>,
88 + /// Pixels per caller unit, e.g. 2 to pass coordinates in Retina points.
89 + #[arg(long, default_value_t = HostInspectScale::PIXELS, value_parser = parse_scale)]
90 + scale: HostInspectScale,
91 + #[arg(long)]
92 + json: bool,
93 + },
94 + /// Crop an image to a rect and write the result out as a PNG.
95 + Crop {
96 + #[arg(long)]
97 + image: PathBuf,
98 + /// Crop rect as x,y,width,height in caller units.
99 + #[arg(long, value_parser = parse_rect)]
100 + rect: HostInspectRect,
101 + /// Destination path for the cropped PNG.
102 + #[arg(long)]
103 + out: PathBuf,
104 + #[arg(long, default_value_t = HostInspectScale::PIXELS, value_parser = parse_scale)]
105 + scale: HostInspectScale,
106 + #[arg(long)]
107 + json: bool,
108 + },
109 + /// Read one line of an image as a run of colors.
110 + Probe {
111 + #[arg(long)]
112 + image: PathBuf,
113 + /// Probe a horizontal line: the row at --position.
114 + #[arg(long, conflicts_with = "column")]
115 + row: bool,
116 + /// Probe a vertical line: the column at --position.
117 + #[arg(long, conflicts_with = "row")]
118 + column: bool,
119 + /// Row or column to read, in caller units.
120 + #[arg(long)]
121 + position: f64,
122 + /// First unit of the probed line; defaults to the image edge.
123 + #[arg(long)]
124 + from: Option<f64>,
125 + /// Last unit of the probed line; defaults to the image edge.
126 + #[arg(long)]
127 + to: Option<f64>,
128 + #[arg(long, default_value_t = HostInspectScale::PIXELS, value_parser = parse_scale)]
129 + scale: HostInspectScale,
130 + #[arg(long)]
131 + json: bool,
132 + },
133 +}
134 +
135 +fn parse_scale(value: &str) -> Result<HostInspectScale> {
136 + value.parse()
137 +}
138 +
139 +fn parse_rect(value: &str) -> Result<HostInspectRect> {
140 + value.parse()
141 +}
142 +
143 +/// Resolve the probe direction. Clap has already rejected `--row --column`;
144 +/// this covers the remaining case of neither flag being given.
145 +fn probe_axis(row: bool, column: bool) -> Result<HostProbeAxis> {
146 + match (row, column) {
147 + (true, false) => Ok(HostProbeAxis::Row),
148 + (false, true) => Ok(HostProbeAxis::Column),
149 + _ => Err(anyhow!("probe needs exactly one of --row or --column")),
150 + }
151 +}
152 +
153 #[derive(Debug, Subcommand)]
154 enum SimulatorCommand {
155 Find {
@@ -111,6 +200,91 @@ async fn main() -> Result<()> {
200 AndroidDebugBridge::new().stop_androidkit(&serial)?;
201 println!("Stopped XCRS AndroidKit on {serial}");
202 }
203 + Commands::Host(HostCommand::Windows { owner, json }) => {
204 + let windows = HostInspect::new().windows(owner.as_deref())?;
205 + if json {
206 + println!("{}", serde_json::to_string_pretty(&windows)?);
207 + } else if windows.windows.is_empty() {
208 + println!("No on-screen windows");
209 + } else {
210 + for window in windows.windows {
211 + println!(
212 + "{} {} {} {}",
213 + window.id, window.owner, window.title, window.bounds
214 + );
215 + }
216 + }
217 + }
218 + Commands::Host(HostCommand::Ocr {
219 + image,
220 + languages,
221 + scale,
222 + json,
223 + }) => {
224 + let result = HostInspect::new().recognize_text(&HostOcrRequest {
225 + image,
226 + scale,
227 + languages,
228 + })?;
229 + if json {
230 + println!("{}", serde_json::to_string_pretty(&result)?);
231 + } else if result.results.is_empty() {
232 + println!("No text found in {}", result.image.path);
233 + } else {
234 + for recognized in result.results {
235 + println!("{:.2} {}", recognized.confidence, recognized.text);
236 + }
237 + }
238 + }
239 + Commands::Host(HostCommand::Crop {
240 + image,
241 + rect,
242 + out,
243 + scale,
244 + json,
245 + }) => {
246 + let result = HostInspect::new().crop(&HostCropRequest {
247 + image,
248 + rect,
249 + out,
250 + scale,
251 + })?;
252 + if json {
253 + println!("{}", serde_json::to_string_pretty(&result)?);
254 + } else {
255 + println!(
256 + "Wrote {} ({}x{})",
257 + result.out, result.pixel_width, result.pixel_height
258 + );
259 + }
260 + }
261 + Commands::Host(HostCommand::Probe {
262 + image,
263 + row,
264 + column,
265 + position,
266 + from,
267 + to,
268 + scale,
269 + json,
270 + }) => {
271 + let axis = probe_axis(row, column)?;
272 + let result = HostInspect::new().probe(&HostProbeRequest {
273 + image,
274 + axis,
275 + position,
276 + from,
277 + to,
278 + scale,
279 + })?;
280 + if json {
281 + println!("{}", serde_json::to_string_pretty(&result)?);
282 + } else {
283 + for run in result.runs {
284 + println!("{} {} {} {}", run.start, run.end, run.width, run.color);
285 + }
286 + }
287 + }
288 Commands::Simulator(SimulatorCommand::Find { name, json }) => {
289 let simulator = tools.simctl().find_simulator_by_name(&name)?;
290 if json {
@@ -158,3 +332,180 @@ async fn main() -> Result<()> {
332
333 Ok(())
334 }
335 +
336 +#[cfg(test)]
337 +mod tests {
338 + use super::*;
339 +
340 + fn host(arguments: &[&str]) -> HostCommand {
341 + match Cli::try_parse_from(arguments)
342 + .unwrap_or_else(|error| panic!("failed to parse {arguments:?}: {error}"))
343 + .command
344 + {
345 + Some(Commands::Host(command)) => command,
346 + other => panic!("expected a host command, got {other:?}"),
347 + }
348 + }
349 +
350 + #[test]
351 + fn parses_the_host_windows_command() {
352 + let HostCommand::Windows { owner, json } = host(&["xcrs", "host", "windows", "--json"])
353 + else {
354 + panic!("expected the windows command");
355 + };
356 +
357 + assert_eq!(owner, None);
358 + assert!(json);
359 + }
360 +
361 + #[test]
362 + fn filters_windows_by_owner() {
363 + let HostCommand::Windows { owner, json } =
364 + host(&["xcrs", "host", "windows", "--owner", "Xcode"])
365 + else {
366 + panic!("expected the windows command");
367 + };
368 +
369 + assert_eq!(owner.as_deref(), Some("Xcode"));
370 + assert!(!json);
371 + }
372 +
373 + #[test]
374 + fn defaults_the_host_scale_to_pixels() {
375 + let HostCommand::Ocr {
376 + languages, scale, ..
377 + } = host(&["xcrs", "host", "ocr", "--image", "shot.png"])
378 + else {
379 + panic!("expected the ocr command");
380 + };
381 +
382 + assert!(languages.is_empty());
383 + assert_eq!(scale, HostInspectScale::PIXELS);
384 + }
385 +
386 + #[test]
387 + fn collects_repeated_languages_and_the_scale() {
388 + let HostCommand::Ocr {
389 + image,
390 + languages,
391 + scale,
392 + ..
393 + } = host(&[
394 + "xcrs",
395 + "host",
396 + "ocr",
397 + "--image",
398 + "shot.png",
399 + "--language",
400 + "en-US",
401 + "--language",
402 + "sv-SE",
403 + "--scale",
404 + "2",
405 + ])
406 + else {
407 + panic!("expected the ocr command");
408 + };
409 +
410 + assert_eq!(image, PathBuf::from("shot.png"));
411 + assert_eq!(languages, ["en-US", "sv-SE"]);
412 + assert_eq!(scale.pixels_per_unit(), 2.0);
413 + }
414 +
415 + #[test]
416 + fn parses_the_host_crop_rect() {
417 + let HostCommand::Crop {
418 + image, rect, out, ..
419 + } = host(&[
420 + "xcrs",
421 + "host",
422 + "crop",
423 + "--image",
424 + "shot.png",
425 + "--rect",
426 + "1,2,30,40",
427 + "--out",
428 + "crop.png",
429 + ])
430 + else {
431 + panic!("expected the crop command");
432 + };
433 +
434 + assert_eq!(image, PathBuf::from("shot.png"));
435 + assert_eq!(rect, HostInspectRect::new(1.0, 2.0, 30.0, 40.0));
436 + assert_eq!(out, PathBuf::from("crop.png"));
437 + }
438 +
439 + #[test]
440 + fn rejects_a_crop_rect_that_is_not_four_numbers() {
441 + let error = Cli::try_parse_from([
442 + "xcrs", "host", "crop", "--image", "shot.png", "--rect", "1,2,3", "--out", "crop.png",
443 + ])
444 + .expect_err("a three part rect should not parse");
445 +
446 + assert!(error.to_string().contains("x,y,w,h"), "{error}");
447 + }
448 +
449 + #[test]
450 + fn parses_the_probe_bounds() {
451 + let HostCommand::Probe {
452 + image,
453 + row,
454 + column,
455 + position,
456 + from,
457 + to,
458 + scale,
459 + json,
460 + } = host(&[
461 + "xcrs",
462 + "host",
463 + "probe",
464 + "--image",
465 + "shot.png",
466 + "--row",
467 + "--position",
468 + "12",
469 + "--from",
470 + "4",
471 + "--to",
472 + "20",
473 + ])
474 + else {
475 + panic!("expected the probe command");
476 + };
477 +
478 + assert_eq!(image, PathBuf::from("shot.png"));
479 + assert_eq!(probe_axis(row, column).expect("row"), HostProbeAxis::Row);
480 + assert_eq!(position, 12.0);
481 + assert_eq!(from, Some(4.0));
482 + assert_eq!(to, Some(20.0));
483 + assert_eq!(scale, HostInspectScale::PIXELS);
484 + assert!(!json);
485 + }
486 +
487 + #[test]
488 + fn rejects_a_probe_with_both_axes() {
489 + let error = Cli::try_parse_from([
490 + "xcrs",
491 + "host",
492 + "probe",
493 + "--image",
494 + "shot.png",
495 + "--row",
496 + "--column",
497 + "--position",
498 + "12",
499 + ])
500 + .expect_err("--row and --column should conflict");
501 +
502 + assert_eq!(error.kind(), clap::error::ErrorKind::ArgumentConflict);
503 + }
504 +
505 + #[test]
506 + fn rejects_a_probe_without_an_axis() {
507 + let error = probe_axis(false, false).expect_err("an axis is required");
508 +
509 + assert!(error.to_string().contains("--row or --column"), "{error}");
510 + }
511 +}
crates/xcrs/src/xcrs.rs
+984
@@ -1184,6 +1184,525 @@ fn shell_escape(value: &str) -> String {
1184 format!("'{}'", value.replace('\'', "'\\''"))
1185 }
1186
1187 +/// The name of the Swift helper that reads the host machine.
1188 +const HOST_INSPECT_HELPER: &str = "xcrs-host-inspect";
1189 +
1190 +/// What one `xcrs-host-inspect` run wrote, whether or not it succeeded.
1191 +#[derive(Debug, Clone, PartialEq, Eq)]
1192 +pub struct HostInspectInvocation {
1193 + /// The helper's exit code, or `None` when a signal killed it.
1194 + pub exit_code: Option<i32>,
1195 + pub stdout: String,
1196 + pub stderr: String,
1197 +}
1198 +
1199 +impl HostInspectInvocation {
1200 + pub fn succeeded(&self) -> bool {
1201 + self.exit_code == Some(0)
1202 + }
1203 +
1204 + /// Whether the helper rejected the invocation itself rather than failing
1205 + /// the work it was asked to do. The helper reserves exit code 2 for a
1206 + /// malformed command and 1 for a failed operation.
1207 + pub fn is_usage_error(&self) -> bool {
1208 + self.exit_code == Some(2)
1209 + }
1210 +
1211 + /// The failure text, preferring stderr and falling back to stdout.
1212 + pub fn message(&self) -> String {
1213 + [self.stderr.trim(), self.stdout.trim()]
1214 + .into_iter()
1215 + .filter(|part| !part.is_empty())
1216 + .collect::<Vec<_>>()
1217 + .join("\n")
1218 + }
1219 +}
1220 +
1221 +/// Converts between the units a caller reasons about and image pixels.
1222 +///
1223 +/// A Retina screenshot is two pixels per point, so a scale of `2` accepts and
1224 +/// reports points; the default of `1` accepts and reports pixels.
1225 +#[derive(Debug, Clone, Copy, PartialEq)]
1226 +pub struct HostInspectScale(f64);
1227 +
1228 +impl HostInspectScale {
1229 + /// Pixels, which is the helper's default.
1230 + pub const PIXELS: Self = Self(1.0);
1231 +
1232 + pub fn new(pixels_per_unit: f64) -> Result<Self> {
1233 + if !pixels_per_unit.is_finite() || pixels_per_unit <= 0.0 {
1234 + return Err(anyhow!(
1235 + "scale must be a positive number, got {pixels_per_unit}"
1236 + ));
1237 + }
1238 +
1239 + Ok(Self(pixels_per_unit))
1240 + }
1241 +
1242 + pub fn pixels_per_unit(&self) -> f64 {
1243 + self.0
1244 + }
1245 +
1246 + pub fn pixels(&self, units: f64) -> f64 {
1247 + units * self.0
1248 + }
1249 +
1250 + pub fn units(&self, pixels: f64) -> f64 {
1251 + pixels / self.0
1252 + }
1253 +}
1254 +
1255 +impl Default for HostInspectScale {
1256 + fn default() -> Self {
1257 + Self::PIXELS
1258 + }
1259 +}
1260 +
1261 +impl std::fmt::Display for HostInspectScale {
1262 + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
1263 + write!(formatter, "{}", self.0)
1264 + }
1265 +}
1266 +
1267 +impl std::str::FromStr for HostInspectScale {
1268 + type Err = anyhow::Error;
1269 +
1270 + fn from_str(value: &str) -> Result<Self> {
1271 + let pixels_per_unit = value
1272 + .trim()
1273 + .parse::<f64>()
1274 + .with_context(|| format!("scale expects a number, got {value}"))?;
1275 +
1276 + Self::new(pixels_per_unit)
1277 + }
1278 +}
1279 +
1280 +/// A rectangle in caller units. Rows run top down and columns left to right,
1281 +/// matching the rest of XCRS.
1282 +#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
1283 +pub struct HostInspectRect {
1284 + pub x: f64,
1285 + pub y: f64,
1286 + pub width: f64,
1287 + pub height: f64,
1288 +}
1289 +
1290 +impl HostInspectRect {
1291 + pub fn new(x: f64, y: f64, width: f64, height: f64) -> Self {
1292 + Self {
1293 + x,
1294 + y,
1295 + width,
1296 + height,
1297 + }
1298 + }
1299 +
1300 + pub fn validate(&self) -> Result<()> {
1301 + if [self.x, self.y, self.width, self.height]
1302 + .into_iter()
1303 + .any(|value| !value.is_finite())
1304 + {
1305 + return Err(anyhow!("rect values must be finite numbers, got {self}"));
1306 + }
1307 + if self.width <= 0.0 || self.height <= 0.0 {
1308 + return Err(anyhow!(
1309 + "rect width and height must be positive, got {self}"
1310 + ));
1311 + }
1312 +
1313 + Ok(())
1314 + }
1315 +}
1316 +
1317 +impl std::fmt::Display for HostInspectRect {
1318 + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
1319 + write!(
1320 + formatter,
1321 + "{},{},{},{}",
1322 + self.x, self.y, self.width, self.height
1323 + )
1324 + }
1325 +}
1326 +
1327 +impl std::str::FromStr for HostInspectRect {
1328 + type Err = anyhow::Error;
1329 +
1330 + fn from_str(value: &str) -> Result<Self> {
1331 + let parts = value.split(',').collect::<Vec<_>>();
1332 + if parts.len() != 4 {
1333 + return Err(anyhow!("rect expects x,y,w,h, got {value}"));
1334 + }
1335 +
1336 + let mut numbers = [0.0_f64; 4];
1337 + for (index, part) in parts.into_iter().enumerate() {
1338 + numbers[index] = part
1339 + .trim()
1340 + .parse::<f64>()
1341 + .with_context(|| format!("rect values must be numbers, got {value}"))?;
1342 + }
1343 +
1344 + let rect = Self::new(numbers[0], numbers[1], numbers[2], numbers[3]);
1345 + rect.validate()?;
1346 +
1347 + Ok(rect)
1348 + }
1349 +}
1350 +
1351 +/// The image a result was read from, in both pixels and caller units.
1352 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1353 +#[serde(rename_all = "camelCase")]
1354 +pub struct HostInspectImage {
1355 + pub path: String,
1356 + pub pixel_width: i64,
1357 + pub pixel_height: i64,
1358 + pub scale: f64,
1359 + pub width: f64,
1360 + pub height: f64,
1361 +}
1362 +
1363 +/// An on-screen window in the global display space.
1364 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1365 +pub struct HostWindow {
1366 + pub id: i64,
1367 + pub owner: String,
1368 + pub title: String,
1369 + pub layer: i64,
1370 + /// Screen points, unaffected by the requested scale.
1371 + pub bounds: HostInspectRect,
1372 +}
1373 +
1374 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1375 +pub struct HostWindowList {
1376 + pub windows: Vec<HostWindow>,
1377 +}
1378 +
1379 +/// One recognized text box, ordered top to bottom and then left to right.
1380 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1381 +pub struct HostRecognizedText {
1382 + pub text: String,
1383 + pub confidence: f64,
1384 + pub x: f64,
1385 + pub y: f64,
1386 + pub width: f64,
1387 + pub height: f64,
1388 +}
1389 +
1390 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1391 +pub struct HostOcrResult {
1392 + pub image: HostInspectImage,
1393 + /// The languages the request was configured with; empty means the
1394 + /// platform's own default.
1395 + pub languages: Vec<String>,
1396 + pub results: Vec<HostRecognizedText>,
1397 +}
1398 +
1399 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1400 +#[serde(rename_all = "camelCase")]
1401 +pub struct HostCropResult {
1402 + pub image: HostInspectImage,
1403 + /// The requested rect, in caller units.
1404 + pub rect: HostInspectRect,
1405 + pub out: String,
1406 + pub pixel_width: i64,
1407 + pub pixel_height: i64,
1408 +}
1409 +
1410 +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
1411 +#[serde(rename_all = "lowercase")]
1412 +pub enum HostProbeAxis {
1413 + Row,
1414 + Column,
1415 +}
1416 +
1417 +impl HostProbeAxis {
1418 + /// The flag the helper reads this axis from.
1419 + pub fn flag(&self) -> &'static str {
1420 + match self {
1421 + Self::Row => "--row",
1422 + Self::Column => "--column",
1423 + }
1424 + }
1425 +}
1426 +
1427 +impl std::fmt::Display for HostProbeAxis {
1428 + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
1429 + formatter.write_str(match self {
1430 + Self::Row => "row",
1431 + Self::Column => "column",
1432 + })
1433 + }
1434 +}
1435 +
1436 +/// A stretch of one color along a probed line, half open in caller units.
1437 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1438 +pub struct HostColorRun {
1439 + pub start: f64,
1440 + pub end: f64,
1441 + pub width: f64,
1442 + /// Quantized `#rrggbb`, so low-order noise does not split a surface.
1443 + pub color: String,
1444 +}
1445 +
1446 +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
1447 +pub struct HostProbeResult {
1448 + pub image: HostInspectImage,
1449 + pub axis: HostProbeAxis,
1450 + pub position: f64,
1451 + pub from: f64,
1452 + pub to: f64,
1453 + pub runs: Vec<HostColorRun>,
1454 +}
1455 +
1456 +#[derive(Debug, Clone, PartialEq)]
1457 +pub struct HostOcrRequest {
1458 + pub image: PathBuf,
1459 + pub scale: HostInspectScale,
1460 + /// Empty leaves the platform on its own default.
1461 + pub languages: Vec<String>,
1462 +}
1463 +
1464 +impl HostOcrRequest {
1465 + pub fn new(image: impl Into<PathBuf>) -> Self {
1466 + Self {
1467 + image: image.into(),
1468 + scale: HostInspectScale::PIXELS,
1469 + languages: Vec::new(),
1470 + }
1471 + }
1472 +}
1473 +
1474 +#[derive(Debug, Clone, PartialEq)]
1475 +pub struct HostCropRequest {
1476 + pub image: PathBuf,
1477 + pub rect: HostInspectRect,
1478 + pub out: PathBuf,
1479 + pub scale: HostInspectScale,
1480 +}
1481 +
1482 +impl HostCropRequest {
1483 + pub fn new(image: impl Into<PathBuf>, rect: HostInspectRect, out: impl Into<PathBuf>) -> Self {
1484 + Self {
1485 + image: image.into(),
1486 + rect,
1487 + out: out.into(),
1488 + scale: HostInspectScale::PIXELS,
1489 + }
1490 + }
1491 +}
1492 +
1493 +#[derive(Debug, Clone, PartialEq)]
1494 +pub struct HostProbeRequest {
1495 + pub image: PathBuf,
1496 + pub axis: HostProbeAxis,
1497 + pub position: f64,
1498 + pub from: Option<f64>,
1499 + pub to: Option<f64>,
1500 + pub scale: HostInspectScale,
1501 +}
1502 +
1503 +impl HostProbeRequest {
1504 + pub fn new(image: impl Into<PathBuf>, axis: HostProbeAxis, position: f64) -> Self {
1505 + Self {
1506 + image: image.into(),
1507 + axis,
1508 + position,
1509 + from: None,
1510 + to: None,
1511 + scale: HostInspectScale::PIXELS,
1512 + }
1513 + }
1514 +}
1515 +
1516 +/// Client for the `xcrs-host-inspect` helper, which reads the macOS host on
1517 +/// behalf of XCRS control runs: on-screen windows, image pixels, and text.
1518 +#[derive(Debug, Clone)]
1519 +pub struct HostInspect {
1520 + helper_path: PathBuf,
1521 +}
1522 +
1523 +impl Default for HostInspect {
1524 + fn default() -> Self {
1525 + Self {
1526 + helper_path: discover_host_inspect_path(),
1527 + }
1528 + }
1529 +}
1530 +
1531 +impl HostInspect {
1532 + pub fn new() -> Self {
1533 + Self::default()
1534 + }
1535 +
1536 + pub fn with_path(helper_path: impl Into<PathBuf>) -> Self {
1537 + Self {
1538 + helper_path: helper_path.into(),
1539 + }
1540 + }
1541 +
1542 + pub fn helper_path(&self) -> &Path {
1543 + &self.helper_path
1544 + }
1545 +
1546 + /// Run the helper and report what it wrote.
1547 + ///
1548 + /// A non-zero exit is part of the helper's contract rather than an error
1549 + /// here, so a caller can forward the code instead of re-deriving it. This
1550 + /// only fails when the helper could not be started at all.
1551 + pub fn invoke(&self, arguments: &[String]) -> Result<HostInspectInvocation> {
1552 + let output = Command::new(&self.helper_path)
1553 + .args(arguments)
1554 + .output()
1555 + .with_context(|| {
1556 + format!(
1557 + "failed to run {} (build it with `swift build` in xcrs-controlkit)",
1558 + self.helper_path.display()
1559 + )
1560 + })?;
1561 +
1562 + Ok(HostInspectInvocation {
1563 + exit_code: output.status.code(),
1564 + stdout: String::from_utf8_lossy(&output.stdout).into_owned(),
1565 + stderr: String::from_utf8_lossy(&output.stderr).into_owned(),
1566 + })
1567 + }
1568 +
1569 + /// List the host's on-screen windows, ordered by layer and then id.
1570 + pub fn windows(&self, owner: Option<&str>) -> Result<HostWindowList> {
1571 + let mut arguments = vec!["windows".to_string()];
1572 + if let Some(owner) = owner {
1573 + arguments.push("--owner".to_string());
1574 + arguments.push(owner.to_string());
1575 + }
1576 +
1577 + self.decode(&arguments)
1578 + }
1579 +
1580 + /// Read the text in an image.
1581 + pub fn recognize_text(&self, request: &HostOcrRequest) -> Result<HostOcrResult> {
1582 + let mut arguments = vec![
1583 + "ocr".to_string(),
1584 + "--image".to_string(),
1585 + path_to_string(&request.image)?,
1586 + "--scale".to_string(),
1587 + request.scale.to_string(),
1588 + ];
1589 + for language in &request.languages {
1590 + arguments.push("--language".to_string());
1591 + arguments.push(language.clone());
1592 + }
1593 +
1594 + self.decode(&arguments)
1595 + }
1596 +
1597 + /// Crop an image to a rect and write the result out as a PNG.
1598 + pub fn crop(&self, request: &HostCropRequest) -> Result<HostCropResult> {
1599 + request.rect.validate()?;
1600 + let arguments = vec![
1601 + "crop".to_string(),
1602 + "--image".to_string(),
1603 + path_to_string(&request.image)?,
1604 + "--rect".to_string(),
1605 + request.rect.to_string(),
1606 + "--out".to_string(),
1607 + path_to_string(&request.out)?,
1608 + "--scale".to_string(),
1609 + request.scale.to_string(),
1610 + ];
1611 +
1612 + self.decode(&arguments)
1613 + }
1614 +
1615 + /// Read one line of an image as a run of colors.
1616 + pub fn probe(&self, request: &HostProbeRequest) -> Result<HostProbeResult> {
1617 + if !request.position.is_finite() {
1618 + return Err(anyhow!(
1619 + "probe position must be a finite number, got {}",
1620 + request.position
1621 + ));
1622 + }
1623 +
1624 + let mut arguments = vec![
1625 + "probe".to_string(),
1626 + "--image".to_string(),
1627 + path_to_string(&request.image)?,
1628 + request.axis.flag().to_string(),
1629 + request.position.to_string(),
1630 + "--scale".to_string(),
1631 + request.scale.to_string(),
1632 + ];
1633 + for (flag, bound) in [("--from", request.from), ("--to", request.to)] {
1634 + let Some(bound) = bound else { continue };
1635 + if !bound.is_finite() {
1636 + return Err(anyhow!("probe {flag} must be a finite number, got {bound}"));
1637 + }
1638 + arguments.push(flag.to_string());
1639 + arguments.push(bound.to_string());
1640 + }
1641 +
1642 + self.decode(&arguments)
1643 + }
1644 +
1645 + fn decode<T>(&self, arguments: &[String]) -> Result<T>
1646 + where
1647 + T: serde::de::DeserializeOwned,
1648 + {
1649 + let command = arguments
1650 + .first()
1651 + .map(String::as_str)
1652 + .unwrap_or("invocation");
1653 + let invocation = self.invoke(arguments)?;
1654 + if !invocation.succeeded() {
1655 + let status = match invocation.exit_code {
1656 + Some(code) => format!("exited with status {code}"),
1657 + None => "was killed by a signal".to_string(),
1658 + };
1659 + let message = invocation.message();
1660 +
1661 + return Err(if message.is_empty() {
1662 + anyhow!("xcrs-host-inspect {command} {status}")
1663 + } else {
1664 + anyhow!("xcrs-host-inspect {command} {status}: {message}")
1665 + });
1666 + }
1667 +
1668 + serde_json::from_str(&invocation.stdout)
1669 + .with_context(|| format!("failed to decode xcrs-host-inspect {command} JSON"))
1670 + }
1671 +}
1672 +
1673 +fn discover_host_inspect_path() -> PathBuf {
1674 + let executable = if cfg!(windows) {
1675 + format!("{HOST_INSPECT_HELPER}.exe")
1676 + } else {
1677 + HOST_INSPECT_HELPER.to_string()
1678 + };
1679 +
1680 + let mut candidates = Vec::new();
1681 + if let Some(path) = env::var_os("XCRS_HOST_INSPECT") {
1682 + candidates.push(PathBuf::from(path));
1683 + }
1684 + // A release ships the helper next to the `xcrs` binary; a SwiftPM checkout
1685 + // leaves it in the build directory.
1686 + if let Some(directory) = env::current_exe()
1687 + .ok()
1688 + .and_then(|path| path.parent().map(Path::to_path_buf))
1689 + {
1690 + candidates.push(directory.join(&executable));
1691 + }
1692 + candidates.push(Path::new(".build").join("release").join(&executable));
1693 + candidates.push(Path::new(".build").join("debug").join(&executable));
1694 + if let Some(home) = env::var_os("HOME").map(PathBuf::from) {
1695 + candidates.push(home.join("bin").join(&executable));
1696 + }
1697 + candidates.push(Path::new("/usr/local/bin").join(&executable));
1698 + candidates.push(Path::new("/opt/homebrew/bin").join(&executable));
1699 +
1700 + candidates
1701 + .into_iter()
1702 + .find(|path| path.is_file())
1703 + .unwrap_or_else(|| PathBuf::from(executable))
1704 +}
1705 +
1706 #[derive(Debug, Clone)]
1707 pub enum XcodeProject {
1708 Project(PathBuf),
@@ -1882,4 +2401,469 @@ SERIAL456 no permissions (user denied) usb:0-2 transport_id:3
2401 // Falls back to the serial when no product/model/device property is reported.
2402 assert_eq!(normalized.name, "R58N123ABCD");
2403 }
2404 +
2405 + #[test]
2406 + fn defaults_host_inspect_scale_to_pixels() {
2407 + let scale = HostInspectScale::default();
2408 +
2409 + assert_eq!(scale, HostInspectScale::PIXELS);
2410 + assert_eq!(scale.pixels_per_unit(), 1.0);
2411 + assert_eq!(scale.to_string(), "1");
2412 + }
2413 +
2414 + #[test]
2415 + fn converts_between_units_and_pixels() {
2416 + let retina = HostInspectScale::new(2.0).expect("2 is a valid scale");
2417 +
2418 + assert_eq!(retina.pixels(720.0), 1440.0);
2419 + assert_eq!(retina.units(1440.0), 720.0);
2420 + assert_eq!(retina.to_string(), "2");
2421 + }
2422 +
2423 + #[test]
2424 + fn parses_host_inspect_scale() {
2425 + let scale = "2.5".parse::<HostInspectScale>().expect("2.5 should parse");
2426 +
2427 + assert_eq!(scale.pixels_per_unit(), 2.5);
2428 + assert!("0".parse::<HostInspectScale>().is_err());
2429 + assert!("-1".parse::<HostInspectScale>().is_err());
2430 + assert!("nan".parse::<HostInspectScale>().is_err());
2431 + assert!("two".parse::<HostInspectScale>().is_err());
2432 + }
2433 +
2434 + #[test]
2435 + fn parses_and_formats_host_inspect_rect() {
2436 + let rect = "10, 20, 30, 40"
2437 + .parse::<HostInspectRect>()
2438 + .expect("a four number rect should parse");
2439 +
2440 + assert_eq!(rect, HostInspectRect::new(10.0, 20.0, 30.0, 40.0));
2441 + assert_eq!(rect.to_string(), "10,20,30,40");
2442 + assert_eq!(
2443 + rect.to_string()
2444 + .parse::<HostInspectRect>()
2445 + .expect("round trip"),
2446 + rect
2447 + );
2448 + }
2449 +
2450 + #[test]
2451 + fn rejects_a_rect_that_cannot_be_used() {
2452 + assert!("0,0,4".parse::<HostInspectRect>().is_err());
2453 + assert!("0,0,4,4,4".parse::<HostInspectRect>().is_err());
2454 + assert!("a,b,c,d".parse::<HostInspectRect>().is_err());
2455 + assert!("0,0,0,4".parse::<HostInspectRect>().is_err());
2456 + assert!("0,0,4,-1".parse::<HostInspectRect>().is_err());
2457 + assert!("0,0,inf,4".parse::<HostInspectRect>().is_err());
2458 + }
2459 +
2460 + #[test]
2461 + fn discovers_the_host_inspect_helper_from_the_environment() {
2462 + let directory = tempfile::tempdir().expect("a temporary directory");
2463 + let helper = directory.path().join("inspect-from-environment");
2464 + std::fs::write(&helper, "#!/bin/sh\n").expect("the helper should write");
2465 +
2466 + let previous = env::var_os("XCRS_HOST_INSPECT");
2467 + env::set_var("XCRS_HOST_INSPECT", &helper);
2468 + let discovered = discover_host_inspect_path();
2469 + match previous {
2470 + Some(value) => env::set_var("XCRS_HOST_INSPECT", value),
2471 + None => env::remove_var("XCRS_HOST_INSPECT"),
2472 + }
2473 +
2474 + // The environment wins over every other candidate, exactly as it does
2475 + // for the Android debug bridge.
2476 + assert_eq!(discovered, helper);
2477 + }
2478 +
2479 + /// Writes a stand-in for the Swift helper so the client can be exercised
2480 + /// without a built `xcrs-host-inspect` or a macOS host.
2481 + #[cfg(unix)]
2482 + fn write_fake_helper(directory: &Path, script: &str) -> PathBuf {
2483 + use std::os::unix::fs::PermissionsExt;
2484 +
2485 + let path = directory.join(HOST_INSPECT_HELPER);
2486 + std::fs::write(&path, script).expect("the fake helper should write");
2487 + let mut permissions = std::fs::metadata(&path)
2488 + .expect("the fake helper should stat")
2489 + .permissions();
2490 + permissions.set_mode(0o755);
2491 + std::fs::set_permissions(&path, permissions).expect("the fake helper should be executable");
2492 +
2493 + path
2494 + }
2495 +
2496 + /// A fake helper that records the arguments it was handed and prints
2497 + /// `stdout` before exiting with `exit_code`.
2498 + #[cfg(unix)]
2499 + fn recording_helper(directory: &Path, stdout: &str, exit_code: i32) -> (PathBuf, PathBuf) {
2500 + let recorded = directory.join("arguments.txt");
2501 + let script = format!(
2502 + "#!/bin/sh\nprintf '%s\\n' \"$@\" > '{}'\ncat <<'EOF'\n{}\nEOF\nexit {}\n",
2503 + recorded.display(),
2504 + stdout,
2505 + exit_code
2506 + );
2507 +
2508 + (write_fake_helper(directory, &script), recorded)
2509 + }
2510 +
2511 + #[cfg(unix)]
2512 + fn recorded_arguments(recorded: &Path) -> Vec<String> {
2513 + std::fs::read_to_string(recorded)
2514 + .expect("the helper should have recorded its arguments")
2515 + .lines()
2516 + .map(str::to_string)
2517 + .collect()
2518 + }
2519 +
2520 + #[cfg(unix)]
2521 + const WINDOW_LIST_JSON: &str = r#"{
2522 + "windows": [
2523 + {
2524 + "id": 42,
2525 + "owner": "Safari",
2526 + "title": "Example",
2527 + "layer": 0,
2528 + "bounds": { "x": 0, "y": 25, "width": 1440, "height": 875 }
2529 + }
2530 + ]
2531 + }"#;
2532 +
2533 + #[cfg(unix)]
2534 + const IMAGE_JSON: &str = r#"{
2535 + "path": "/tmp/screenshot.png",
2536 + "pixelWidth": 2880,
2537 + "pixelHeight": 1800,
2538 + "scale": 2,
2539 + "width": 1440,
2540 + "height": 900
2541 + }"#;
2542 +
2543 + #[cfg(unix)]
2544 + #[test]
2545 + fn decodes_a_host_window_list() {
2546 + let directory = tempfile::tempdir().expect("a temporary directory");
2547 + let (helper, _) = recording_helper(directory.path(), WINDOW_LIST_JSON, 0);
2548 + let host = HostInspect::with_path(helper);
2549 +
2550 + let list = host.windows(None).expect("the window list should decode");
2551 +
2552 + assert_eq!(list.windows.len(), 1);
2553 + let window = &list.windows[0];
2554 + assert_eq!(window.id, 42);
2555 + assert_eq!(window.owner, "Safari");
2556 + assert_eq!(window.title, "Example");
2557 + assert_eq!(window.layer, 0);
2558 + // Bounds stay in screen points, so they are not scaled.
2559 + assert_eq!(
2560 + window.bounds,
2561 + HostInspectRect::new(0.0, 25.0, 1440.0, 875.0)
2562 + );
2563 + }
2564 +
2565 + #[cfg(unix)]
2566 + #[test]
2567 + fn builds_the_windows_arguments() {
2568 + let directory = tempfile::tempdir().expect("a temporary directory");
2569 + let (helper, recorded) = recording_helper(directory.path(), WINDOW_LIST_JSON, 0);
2570 + let host = HostInspect::with_path(helper);
2571 +
2572 + host.windows(None).expect("windows should succeed");
2573 + assert_eq!(recorded_arguments(&recorded), vec!["windows"]);
2574 +
2575 + host.windows(Some("Safari"))
2576 + .expect("a filtered windows call should succeed");
2577 + assert_eq!(
2578 + recorded_arguments(&recorded),
2579 + vec!["windows", "--owner", "Safari"]
2580 + );
2581 + }
2582 +
2583 + #[cfg(unix)]
2584 + #[test]
2585 + fn builds_the_ocr_arguments_and_decodes_the_result() {
2586 + let directory = tempfile::tempdir().expect("a temporary directory");
2587 + let stdout = format!(
2588 + r#"{{
2589 + "image": {},
2590 + "languages": ["en-US", "sv-SE"],
2591 + "results": [
2592 + {{
2593 + "text": "Sign in",
2594 + "confidence": 0.9,
2595 + "x": 10,
2596 + "y": 20,
2597 + "width": 100,
2598 + "height": 30
2599 + }}
2600 + ]
2601 + }}"#,
2602 + IMAGE_JSON
2603 + );
2604 + let (helper, recorded) = recording_helper(directory.path(), &stdout, 0);
2605 + let host = HostInspect::with_path(helper);
2606 +
2607 + let mut request = HostOcrRequest::new("/tmp/screenshot.png");
2608 + request.scale = HostInspectScale::new(2.0).expect("2 is a valid scale");
2609 + request.languages = vec!["en-US".to_string(), "sv-SE".to_string()];
2610 +
2611 + let result = host
2612 + .recognize_text(&request)
2613 + .expect("the OCR result should decode");
2614 +
2615 + assert_eq!(
2616 + recorded_arguments(&recorded),
2617 + vec![
2618 + "ocr",
2619 + "--image",
2620 + "/tmp/screenshot.png",
2621 + "--scale",
2622 + "2",
2623 + "--language",
2624 + "en-US",
2625 + "--language",
2626 + "sv-SE",
2627 + ]
2628 + );
2629 + // Image geometry is reported in caller units beside the pixel size.
2630 + assert_eq!(result.image.pixel_width, 2880);
2631 + assert_eq!(result.image.width, 1440.0);
2632 + assert_eq!(result.image.scale, 2.0);
2633 + assert_eq!(result.languages, vec!["en-US", "sv-SE"]);
2634 + assert_eq!(result.results.len(), 1);
2635 + assert_eq!(result.results[0].text, "Sign in");
2636 + assert_eq!(result.results[0].confidence, 0.9);
2637 + }
2638 +
2639 + #[cfg(unix)]
2640 + #[test]
2641 + fn builds_the_crop_arguments_and_decodes_the_result() {
2642 + let directory = tempfile::tempdir().expect("a temporary directory");
2643 + let stdout = format!(
2644 + r#"{{
2645 + "image": {},
2646 + "rect": {{ "x": 0, "y": 0, "width": 4, "height": 4 }},
2647 + "out": "/tmp/crop.png",
2648 + "pixelWidth": 8,
2649 + "pixelHeight": 8
2650 + }}"#,
2651 + IMAGE_JSON
2652 + );
2653 + let (helper, recorded) = recording_helper(directory.path(), &stdout, 0);
2654 + let host = HostInspect::with_path(helper);
2655 + let request = HostCropRequest::new(
2656 + "/tmp/screenshot.png",
2657 + HostInspectRect::new(0.0, 0.0, 4.0, 4.0),
2658 + "/tmp/crop.png",
2659 + );
2660 +
2661 + let result = host.crop(&request).expect("the crop result should decode");
2662 +
2663 + assert_eq!(
2664 + recorded_arguments(&recorded),
2665 + vec![
2666 + "crop",
2667 + "--image",
2668 + "/tmp/screenshot.png",
2669 + "--rect",
2670 + "0,0,4,4",
2671 + "--out",
2672 + "/tmp/crop.png",
2673 + "--scale",
2674 + "1",
2675 + ]
2676 + );
2677 + assert_eq!(result.rect, HostInspectRect::new(0.0, 0.0, 4.0, 4.0));
2678 + assert_eq!(result.out, "/tmp/crop.png");
2679 + assert_eq!(result.pixel_width, 8);
2680 + assert_eq!(result.pixel_height, 8);
2681 + }
2682 +
2683 + #[cfg(unix)]
2684 + #[test]
2685 + fn builds_the_probe_arguments_and_decodes_the_result() {
2686 + let directory = tempfile::tempdir().expect("a temporary directory");
2687 + let stdout = format!(
2688 + r##"{{
2689 + "image": {},
2690 + "axis": "column",
2691 + "position": 1,
2692 + "from": 4,
2693 + "to": 8,
2694 + "runs": [
2695 + {{ "start": 4, "end": 8, "width": 4, "color": "#0000f8" }}
2696 + ]
2697 + }}"##,
2698 + IMAGE_JSON
2699 + );
2700 + let (helper, recorded) = recording_helper(directory.path(), &stdout, 0);
2701 + let host = HostInspect::with_path(helper);
2702 +
2703 + let mut request = HostProbeRequest::new("/tmp/screenshot.png", HostProbeAxis::Column, 1.0);
2704 + request.from = Some(4.0);
2705 + request.to = Some(8.0);
2706 +
2707 + let result = host
2708 + .probe(&request)
2709 + .expect("the probe result should decode");
2710 +
2711 + assert_eq!(
2712 + recorded_arguments(&recorded),
2713 + vec![
2714 + "probe",
2715 + "--image",
2716 + "/tmp/screenshot.png",
2717 + "--column",
2718 + "1",
2719 + "--scale",
2720 + "1",
2721 + "--from",
2722 + "4",
2723 + "--to",
2724 + "8",
2725 + ]
2726 + );
2727 + assert_eq!(result.axis, HostProbeAxis::Column);
2728 + assert_eq!(result.position, 1.0);
2729 + assert_eq!(result.from, 4.0);
2730 + assert_eq!(result.to, 8.0);
2731 + assert_eq!(result.runs.len(), 1);
2732 + assert_eq!(result.runs[0].color, "#0000f8");
2733 + assert_eq!(result.runs[0].start, 4.0);
2734 + assert_eq!(result.runs[0].end, 8.0);
2735 + }
2736 +
2737 + #[cfg(unix)]
2738 + #[test]
2739 + fn omits_the_unset_probe_bounds() {
2740 + let directory = tempfile::tempdir().expect("a temporary directory");
2741 + let stdout = format!(
2742 + r#"{{ "image": {}, "axis": "row", "position": 1, "from": 0, "to": 8, "runs": [] }}"#,
2743 + IMAGE_JSON
2744 + );
2745 + let (helper, recorded) = recording_helper(directory.path(), &stdout, 0);
2746 + let host = HostInspect::with_path(helper);
2747 + let request = HostProbeRequest::new("/tmp/screenshot.png", HostProbeAxis::Row, 1.0);
2748 +
2749 + host.probe(&request).expect("the probe should succeed");
2750 +
2751 + assert_eq!(
2752 + recorded_arguments(&recorded),
2753 + vec![
2754 + "probe",
2755 + "--image",
2756 + "/tmp/screenshot.png",
2757 + "--row",
2758 + "1",
2759 + "--scale",
2760 + "1",
2761 + ]
2762 + );
2763 + }
2764 +
2765 + #[cfg(unix)]
2766 + #[test]
2767 + fn keeps_the_host_inspect_exit_code_as_data() {
2768 + let directory = tempfile::tempdir().expect("a temporary directory");
2769 + let (helper, _) = recording_helper(directory.path(), "", 2);
2770 + let host = HostInspect::with_path(helper);
2771 +
2772 + let invocation = host
2773 + .invoke(&["windows".to_string()])
2774 + .expect("a usage error is still a completed run");
2775 +
2776 + assert_eq!(invocation.exit_code, Some(2));
2777 + assert!(invocation.is_usage_error());
2778 + assert!(!invocation.succeeded());
2779 + }
2780 +
2781 + #[cfg(unix)]
2782 + #[test]
2783 + fn reports_a_failed_helper_with_its_message() {
2784 + let directory = tempfile::tempdir().expect("a temporary directory");
2785 + let script =
2786 + "#!/bin/sh\necho 'cannot open an image /nonexistent/missing.png' >&2\nexit 1\n";
2787 + let helper = write_fake_helper(directory.path(), script);
2788 + let host = HostInspect::with_path(helper);
2789 +
2790 + let error = host
2791 + .windows(None)
2792 + .expect_err("a failed helper should not decode");
2793 +
2794 + let message = error.to_string();
2795 + assert!(message.contains("xcrs-host-inspect windows exited with status 1"));
2796 + assert!(message.contains("cannot open an image /nonexistent/missing.png"));
2797 + }
2798 +
2799 + #[cfg(unix)]
2800 + #[test]
2801 + fn reports_a_silently_failed_helper_without_a_dangling_message() {
2802 + let directory = tempfile::tempdir().expect("a temporary directory");
2803 + let (helper, _) = recording_helper(directory.path(), "", 1);
2804 + let host = HostInspect::with_path(helper);
2805 +
2806 + let error = host
2807 + .windows(None)
2808 + .expect_err("a failed helper should not decode");
2809 +
2810 + assert_eq!(
2811 + error.to_string(),
2812 + "xcrs-host-inspect windows exited with status 1"
2813 + );
2814 + }
2815 +
2816 + #[cfg(unix)]
2817 + #[test]
2818 + fn rejects_output_that_is_not_json() {
2819 + let directory = tempfile::tempdir().expect("a temporary directory");
2820 + let (helper, _) = recording_helper(directory.path(), "not json", 0);
2821 + let host = HostInspect::with_path(helper);
2822 +
2823 + let error = host
2824 + .windows(None)
2825 + .expect_err("malformed output should not decode");
2826 +
2827 + assert!(error
2828 + .to_string()
2829 + .contains("failed to decode xcrs-host-inspect windows JSON"));
2830 + }
2831 +
2832 + #[test]
2833 + fn reports_a_missing_host_inspect_helper() {
2834 + let host = HostInspect::with_path("/nonexistent/xcrs-host-inspect");
2835 +
2836 + let error = host
2837 + .windows(None)
2838 + .expect_err("a missing helper should not run");
2839 +
2840 + let message = error.to_string();
2841 + assert!(message.contains("failed to run /nonexistent/xcrs-host-inspect"));
2842 + assert!(message.contains("swift build"));
2843 + }
2844 +
2845 + #[test]
2846 + fn validates_before_running_the_helper() {
2847 + // Point at a helper that cannot exist, so any error other than the
2848 + // validation one would prove the request was sent anyway.
2849 + let host = HostInspect::with_path("/nonexistent/xcrs-host-inspect");
2850 +
2851 + let error = host
2852 + .crop(&HostCropRequest::new(
2853 + "/tmp/screenshot.png",
2854 + HostInspectRect::new(0.0, 0.0, 0.0, 4.0),
2855 + "/tmp/crop.png",
2856 + ))
2857 + .expect_err("a zero width rect should be rejected");
2858 + assert!(error.to_string().contains("must be positive"));
2859 +
2860 + let error = host
2861 + .probe(&HostProbeRequest::new(
2862 + "/tmp/screenshot.png",
2863 + HostProbeAxis::Row,
2864 + f64::NAN,
2865 + ))
2866 + .expect_err("a non-finite position should be rejected");
2867 + assert!(error.to_string().contains("must be a finite number"));
2868 + }
2869 }
docs/controlkit.md
+98
@@ -158,6 +158,104 @@ Platform-specific operations intentionally reject unsupported input. For
158 example, visionOS rejects raw touch, watchOS rejects pointer and spatial input,
159 and macOS rejects touch and Home-button requests.
160
161 +## Inspect the macOS host
162 +
163 +The `host` subcommands describe what is actually on the local Mac's screen.
164 +They are for the cases where a UI test's own output is not enough: which
165 +windows are up, what text a screenshot contains, or which colors run along one
166 +line of pixels. Unlike the runner tools above they talk to no device and start
167 +no XCTest — they run a small Swift helper,
168 +[`xcrs-host-inspect`](https://github.com/ondeinference/xcrs-controlkit), in
169 +process on the host. They need macOS 13 or later for the AppKit window server
170 +and Vision's text recognizer, are CLI-only, and are not part of the `--mcp`
171 +tool set.
172 +
173 +### Build the helper
174 +
175 +Build it once with SwiftPM from the `xcrs-controlkit` checkout:
176 +
177 +```sh
178 +cd <path-to-xcrs-controlkit>
179 +swift build -c release
180 +```
181 +
182 +The CLI uses the first `xcrs-host-inspect` it finds, in this order:
183 +
184 +1. the path in the `XCRS_HOST_INSPECT` environment variable,
185 +2. next to the `xcrs` binary itself, for a release that ships the helper,
186 +3. `.build/release`, then `.build/debug`, relative to the current directory,
187 +4. `~/bin`, `/usr/local/bin`, and `/opt/homebrew/bin`.
188 +
189 +Because steps 3 are relative, running `xcrs` from inside the checkout finds a
190 +`swift build` without any further setup. Set `XCRS_HOST_INSPECT` when you have
191 +several builds and want to choose between them.
192 +
193 +### Read the screen
194 +
195 +List the host's on-screen windows, ordered by layer and then window id:
196 +
197 +```sh
198 +xcrs host windows
199 +xcrs host windows --owner Mail --json
200 +```
201 +
202 +Each window reports its `id`, the owning application name (`owner`), its
203 +`title`, its `layer`, and its `bounds`. `--owner` keeps only windows owned by
204 +one application name.
205 +
206 +Read the text in an image with Vision and print each string with its
207 +confidence:
208 +
209 +```sh
210 +xcrs host ocr --image shot.png
211 +xcrs host ocr --image shot.png --language en-US --language sv-SE --json
212 +```
213 +
214 +`--language` is repeatable; omit it to use the platform's own default.
215 +
216 +Crop a rect out of an image and write it to a new PNG:
217 +
218 +```sh
219 +xcrs host crop --image shot.png --rect 20,40,320,180 --out header.png
220 +```
221 +
222 +`--rect` takes `x,y,width,height`. The result reports the written path and its
223 +pixel dimensions, and the input image is never modified.
224 +
225 +Probe one row or column and describe it as a run of colors, so a solid surface
226 +collapses to a single line instead of thousands of pixels:
227 +
228 +```sh
229 +xcrs host probe --image shot.png --row --position 300
230 +xcrs host probe --image shot.png --column --position 120 --from 40 --to 260
231 +```
232 +
233 +`--position` is the row or column to read, and exactly one of `--row` or
234 +`--column` is required. `--from` and `--to` clip the line to part of its
235 +length. Runs report `start`, `end`, `width`, and a quantized `#rrggbb`
236 +`color`, so low-order noise does not split one surface into several runs.
237 +
238 +### Coordinates and scale
239 +
240 +Coordinates you pass are in caller units, and `--scale` says how many pixels
241 +one unit covers: the default `1` reads them as pixels, and `2` reads them as
242 +Retina points on a 2x display. `crop` and `probe` honor it for their rect,
243 +position, and bounds. Output geometry follows the same convention — an image
244 +reports `width`/`height` in caller units alongside `pixel_width`/
245 +`pixel_height` — while window bounds are screen points regardless of
246 +`--scale`.
247 +
248 +### Exit codes and errors
249 +
250 +The helper separates a malformed command from failed work: it exits with
251 +status 2 when it rejects the arguments, and 1 when the operation itself fails,
252 +writing the reason to stderr. The CLI validates rects, the scale, and probe
253 +bounds before it spawns the helper, so bad input usually never reaches it.
254 +When the helper does fail, the CLI reports it with its status — for example
255 +`xcrs-host-inspect crop exited with status 1: …` — and exits non-zero. A
256 +helper that cannot be started at all, such as a missing binary, is reported
257 +with a hint to run `swift build` in `xcrs-controlkit`.
258 +
259 ## See also
260
261 For the product-level overview of end-to-end testing with xcrs and