@sigitcodereview / sigit / commits / d73aac7

feat(acp): accept image attachments in prompts

siGit Code now advertises promptCapabilities.image, so an ACP client offers image attachments, and an image block in a prompt is sent to the model as an OpenAI image_url part with a base64 data URL. ACP fixes that capability for the whole connection, but the model can change on any turn and not every model reads images. Since both model lists are hard-coded here, the answer is known per model: - On-device models are text-only. - Cloud tiers follow IMAGE_TIERS in provider.rs. nova, orbit, apex and flux are text-only at the source. The table mirrors onde-cloud's per-provider one. - A model on the user's own endpoint is passed the image and the endpoint decides. With a text-only model active, the image is left out. A note in the thread tells the user, and a bracketed note in the message tells the model, so it does not answer as if it had seen the picture. The model picker marks the models that read images. A message with images stays in history as content parts. A remote backend whose model cannot read them strips them per request, so switching a thread that holds images to a text-only tier keeps working. That switch is flagged to the user once per session and model. Refs #134 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Seto Elkahfi committed Oct 5, 2026 at 09:36 UTC d73aac7b17cc90eaf4d2b092b7597115074f199b

6 files changed +737 -8

.agents/AGENTS.md

+11

@@ -131,6 +131,17 @@ feeds results back. Neither the loop nor ACP/TUI surfaces depend on a concrete b

131 well-formed ones. A block that doesn't parse (or never closes) is dropped from both the reply

132 and history, and `OpenAiBackend::complete` retries once with a note telling the model the call

133 didn't run. Leaving the raw block in history makes the model invent `<function_results>` later.

134 + Image attachments: ACP fixes `promptCapabilities.image` for the whole connection, while the

135 + model can change on any turn, so the capability is always advertised and the decision is made

136 + per prompt. `InferenceBackend::accepts_images` answers for the active model (on-device: no;

137 + remote: `provider::model_accepts_images`, which reads the hard-coded `IMAGE_TIERS` for

138 + `onde-*` ids and says yes for a user's own endpoint). `images_for_turn` in `main.rs` either

139 + passes the images to `send_message_with_images` or drops them with a note to the user and a

140 + bracketed note to the model. A message with images is stored in history as OpenAI content

141 + parts and stays that way; `OpenAiBackend` strips the images per request (`without_images`)

142 + when its model cannot read them, which is what makes a mid-thread switch to a text-only tier

143 + safe. Read history text through `backend::message_text`, never `content.as_str()`.

144 + `IMAGE_TIERS` mirrors onde-cloud's `IMAGE_CANDIDATES`; update both together.

145 - **`src/provider.rs`** — decides *which* backend serves inference. Resolution order, first match

146 wins: (1) override via `OPENAI_BASE_URL`+`OPENAI_API_KEY` or active profile in

147 `~/.config/sigit/providers.toml`; (2) siGit Code Cloud when logged in; (3) on-device.

CHANGELOG.md

+11

@@ -4,6 +4,17 @@

5 ### Added

7 +- **Images can be attached to a prompt in the editor.** siGit Code now

8 + advertises `promptCapabilities.image`, so an ACP client offers image

9 + attachments. The image goes to the model when the active model reads images.

10 + The model picker marks those models with "reads images". The on-device

11 + models and four cloud tiers (`nova`, `orbit`, `apex`, `flux`) are text-only:

12 + with one of them active the image is left out, a note in the thread says so,

13 + and the model is told an image was withheld. Switching a thread that already

14 + holds images to a text-only model flags that once as well. Cloud tiers need

15 + siGit Code Cloud to accept image parts, which is rolling out separately

16 + (#134).

17 +

18 - **The tool-round cap of a headless run can be set.** `--max-tool-rounds <n>`

19 or `SIGIT_MAX_TOOL_ROUNDS` (1 to 500) replaces the built-in 24 for

20 `sigit run` and `sigit -p`. The flag wins over the variable.

src/backend.rs

+169 -2

@@ -289,6 +289,67 @@ pub fn message_image_count(message: &serde_json::Value) -> usize {

289 /// task never blocks on a slow consumer.

290 pub type TokenSink = tokio::sync::mpsc::UnboundedSender<String>;

291

292 +/// An image attached to a user message: base64 data and its media type, as an

293 +/// ACP client sends it.

294 +#[derive(Debug, Clone, PartialEq, Eq)]

295 +pub struct ImageInput {

296 + pub mime_type: String,

297 + pub data: String,

298 +}

299 +

300 +/// What a model that cannot read images is shown in place of one.

301 +const IMAGE_OMITTED_NOTE: &str = "[image omitted: this model cannot read images]";

302 +

303 +/// A user message in history form. Plain text keeps the string `content` every

304 +/// other message uses; with images it becomes OpenAI's content-part array,

305 +/// text first, each image as a base64 `data:` URL.

306 +fn user_message(text: &str, images: &[ImageInput]) -> serde_json::Value {

307 + if images.is_empty() {

308 + return serde_json::json!({ "role": "user", "content": text });

309 + }

310 + let mut parts = Vec::with_capacity(images.len() + 1);

311 + if !text.is_empty() {

312 + parts.push(serde_json::json!({ "type": "text", "text": text }));

313 + }

314 + for image in images {

315 + parts.push(serde_json::json!({

316 + "type": "image_url",

317 + "image_url": {

318 + "url": format!("data:{};base64,{}", image.mime_type, image.data),

319 + },

320 + }));

321 + }

322 + serde_json::json!({ "role": "user", "content": parts })

323 +}

324 +

325 +/// `history` as a text-only model has to receive it: a message that carries

326 +/// images is flattened back to a string, with a note where each image was.

327 +///

328 +/// History itself keeps the images. A thread can move to a model that reads

329 +/// them (or back to one), so what was attached is not thrown away just because

330 +/// the model active right now cannot use it.

331 +fn without_images(history: &[serde_json::Value]) -> Vec<serde_json::Value> {

332 + history

333 + .iter()

334 + .map(|message| {

335 + let images = message_image_count(message);

336 + if images == 0 {

337 + return message.clone();

338 + }

339 + let mut text = message_text(message);

340 + for _ in 0..images {

341 + if !text.is_empty() {

342 + text.push('\n');

343 + }

344 + text.push_str(IMAGE_OMITTED_NOTE);

345 + }

346 + let mut flattened = message.clone();

347 + flattened["content"] = serde_json::Value::String(text);

348 + flattened

349 + })

350 + .collect()

351 +}

352 +

353 // ── The trait ───────────────────────────────────────────────────────────────────

354

355 /// A swappable inference backend driving siGit Code's agent loop.

@@ -327,6 +388,28 @@ pub trait InferenceBackend: Send + Sync {

388 /// request in the session.

389 async fn record_cancelled_tool_results(&self, results: Vec<ToolResult>);

390

391 + /// Start a turn from a user message that carries images.

392 + ///

393 + /// The default drops the images and sends the text, which is right for a

394 + /// backend that cannot read them. Callers check [`Self::accepts_images`]

395 + /// first so the user can be told, instead of the image vanishing.

396 + async fn send_message_with_images(

397 + &self,

398 + text: &str,

399 + images: &[ImageInput],

400 + tools: &[ToolSpec],

401 + sink: Option<&TokenSink>,

402 + ) -> Result<TurnResult, BackendError> {

403 + let _ = images;

404 + self.send_message_with_tools(text, tools, sink).await

405 + }

406 +

407 + /// Whether the model behind this backend reads images. On-device models do

408 + /// not; a remote one answers from its model id.

409 + fn accepts_images(&self) -> bool {

410 + false

411 + }

412 +

413 /// Whether inference runs over the network (a configured provider) rather

414 /// than on-device. Drives UI labelling so the displayed model can't claim a

415 /// local model while requests actually go to the cloud.

@@ -707,6 +790,10 @@ pub struct OpenAiBackend {

790 http: reqwest::Client,

791 /// The full message list sent on each request (system + turns + tool results).

792 history: Mutex<Vec<serde_json::Value>>,

793 + /// Whether `model` reads images (see `provider::model_accepts_images`).

794 + /// When it does not, images already in `history` are left out of each

795 + /// request rather than sent to an endpoint that would refuse them.

796 + accepts_images: bool,

797 }

798

799 impl OpenAiBackend {

@@ -723,10 +810,12 @@ impl OpenAiBackend {

810 if let Some(prompt) = system_prompt {

811 history.push(serde_json::json!({ "role": "system", "content": prompt }));

812 }

813 + let model = model.into();

814 Self {

815 base_url: base_url.into(),

816 api_key: api_key.into(),

729 - model: model.into(),

817 + accepts_images: crate::provider::model_accepts_images(&model),

818 + model,

819 http: reqwest::Client::new(),

820 history: Mutex::new(history),

821 }

@@ -811,9 +900,17 @@ impl OpenAiBackend {

900 let url = format!("{}/chat/completions", self.base_url.trim_end_matches('/'));

901 let streaming = sink.is_some();

902

903 + let messages = {

904 + let history = self.history.lock().await;

905 + if self.accepts_images {

906 + history.clone()

907 + } else {

908 + without_images(&history)

909 + }

910 + };

911 let mut body = serde_json::json!({

912 "model": self.model,

816 - "messages": *self.history.lock().await,

913 + "messages": messages,

914 "stream": streaming,

915 });

916 if allow_tool_calls && !tools.is_empty() {

@@ -1379,6 +1476,21 @@ impl InferenceBackend for OpenAiBackend {

1476 }

1477 }

1478

1479 + async fn send_message_with_images(

1480 + &self,

1481 + text: &str,

1482 + images: &[ImageInput],

1483 + tools: &[ToolSpec],

1484 + sink: Option<&TokenSink>,

1485 + ) -> Result<TurnResult, BackendError> {

1486 + self.history.lock().await.push(user_message(text, images));

1487 + self.complete(tools, true, sink).await

1488 + }

1489 +

1490 + fn accepts_images(&self) -> bool {

1491 + self.accepts_images

1492 + }

1493 +

1494 fn is_remote(&self) -> bool {

1495 true

1496 }

@@ -2114,6 +2226,61 @@ mod tests {

2226 })

2227 }

2228

2229 + #[test]

2230 + fn user_message_uses_content_parts_only_when_there_is_an_image() {

2231 + assert_eq!(

2232 + user_message("hello", &[]),

2233 + serde_json::json!({ "role": "user", "content": "hello" })

2234 + );

2235 +

2236 + let image = ImageInput {

2237 + mime_type: "image/png".to_string(),

2238 + data: "AAAA".to_string(),

2239 + };

2240 + assert_eq!(

2241 + user_message("what is this?", std::slice::from_ref(&image)),

2242 + serde_json::json!({

2243 + "role": "user",

2244 + "content": [

2245 + { "type": "text", "text": "what is this?" },

2246 + { "type": "image_url", "image_url": { "url": "data:image/png;base64,AAAA" } },

2247 + ],

2248 + })

2249 + );

2250 + // An image with no text sends no empty text part.

2251 + let only_image = user_message("", &[image]);

2252 + assert_eq!(only_image["content"].as_array().unwrap().len(), 1);

2253 + }

2254 +

2255 + #[test]

2256 + fn a_text_only_model_is_sent_a_note_in_place_of_each_image() {

2257 + let history = vec![

2258 + serde_json::json!({ "role": "system", "content": "prompt" }),

2259 + user_with_image("what is this?", 16),

2260 + serde_json::json!({ "role": "assistant", "content": "a cat" }),

2261 + ];

2262 + let sent = without_images(&history);

2263 +

2264 + assert_eq!(sent[0], history[0]);

2265 + assert_eq!(sent[2], history[2]);

2266 + assert_eq!(

2267 + sent[1]["content"],

2268 + format!("what is this?\n{IMAGE_OMITTED_NOTE}")

2269 + );

2270 + // History keeps the image for a model that can read it later.

2271 + assert_eq!(message_image_count(&history[1]), 1);

2272 + }

2273 +

2274 + #[test]

2275 + fn a_remote_backend_reads_images_by_model_id() {

2276 + let own_endpoint = OpenAiBackend::new("http://localhost", "", "gpt-4o-mini", None);

2277 + assert!(own_endpoint.accepts_images());

2278 + let text_tier = OpenAiBackend::new("http://localhost", "", "onde-nova", None);

2279 + assert!(!text_tier.accepts_images());

2280 + let image_tier = OpenAiBackend::new("http://localhost", "", "onde-large", None);

2281 + assert!(image_tier.accepts_images());

2282 + }

2283 +

2284 #[test]

2285 fn message_text_reads_both_content_shapes() {

2286 assert_eq!(

src/main.rs

+172 -6

@@ -76,8 +76,8 @@ use agent_client_protocol::schema::v1::{

76 ForkSessionResponse, Implementation, InitializeRequest, InitializeResponse,

77 ListSessionsRequest, ListSessionsResponse, LoadSessionRequest, LoadSessionResponse,

78 McpCapabilities, McpServer, Meta, NewSessionRequest, NewSessionResponse, PermissionOption,

79 - PermissionOptionKind, Plan, PlanEntry, PlanEntryPriority, PlanEntryStatus, PromptRequest,

80 - PromptResponse, RequestPermissionOutcome, RequestPermissionRequest,

79 + PermissionOptionKind, Plan, PlanEntry, PlanEntryPriority, PlanEntryStatus, PromptCapabilities,

80 + PromptRequest, PromptResponse, RequestPermissionOutcome, RequestPermissionRequest,

81 SessionAdditionalDirectoriesCapabilities, SessionCapabilities, SessionConfigOption,

82 SessionConfigOptionCategory, SessionConfigSelectOption, SessionConfigValueId,

83 SessionForkCapabilities, SessionId, SessionInfo, SessionListCapabilities, SessionNotification,

@@ -89,8 +89,8 @@ use agent_client_protocol::{Agent, ByteStreams, Client, ConnectionTo, Responder}

89 use onde::inference::{ChatEngine, GgufModelConfig};

91 use crate::backend::{

92 - InferenceBackend, LocalBackend, OpenAiBackend, ToolResult as BackendToolResult, ToolSpec,

93 - TurnResult,

92 + ImageInput, InferenceBackend, LocalBackend, OpenAiBackend, ToolResult as BackendToolResult,

93 + ToolSpec, TurnResult,

94 };

95 use std::path::PathBuf;

96 use std::sync::atomic::{AtomicBool, Ordering};

@@ -1476,6 +1476,76 @@ impl SiGitAgent {

1476 }

1477 }

1478

1479 +/// Session and model pairs already told that earlier images are hidden, so

1480 +/// the note about a model switch is shown once and not on every prompt.

1481 +static IMAGE_GAP_NOTED: std::sync::Mutex<Vec<(String, String)>> = std::sync::Mutex::new(Vec::new());

1482 +

1483 +/// Decide what happens to a prompt's images for the model about to answer.

1484 +///

1485 +/// The editor can attach an image whatever model is selected, because ACP

1486 +/// fixes `promptCapabilities.image` for the whole connection. Returns the text

1487 +/// and images to send, plus a note for the user when something is being left

1488 +/// out:

1489 +///

1490 +/// - the model cannot read images and the prompt has some: they are dropped,

1491 +/// the user is told, and the model is told too, so it does not answer as if

1492 +/// it had seen them;

1493 +/// - the model cannot read images and the thread has some from before a model

1494 +/// switch (`earlier_images`): the user is told once per session and model.

1495 +fn images_for_turn(

1496 + mut user_text: String,

1497 + images: Vec<ImageInput>,

1498 + accepts_images: bool,

1499 + model_name: &str,

1500 + earlier_images: usize,

1501 + session_id: &str,

1502 +) -> (String, Vec<ImageInput>, Option<String>) {

1503 + if accepts_images {

1504 + return (user_text, images, None);

1505 + }

1506 +

1507 + const HINT: &str = "Models marked \"reads images\" in the model picker can.";

1508 + let plural = |count: usize| if count == 1 { "image" } else { "images" };

1509 +

1510 + if !images.is_empty() {

1511 + let count = images.len();

1512 + if !user_text.is_empty() {

1513 + user_text.push_str("\n\n");

1514 + }

1515 + user_text.push_str(&format!(

1516 + "[The user attached {count} {}. This model cannot read images, so it was not \

1517 + included. Say so if the request depends on it.]",

1518 + plural(count)

1519 + ));

1520 + let notice = format!(

1521 + "> **Image not sent.** {model_name} cannot read images, so the {count} attached {} \

1522 + left out of this message. {HINT}",

1523 + if count == 1 {

1524 + "image was"

1525 + } else {

1526 + "images were"

1527 + }

1528 + );

1529 + return (user_text, Vec::new(), Some(notice));

1530 + }

1531 +

1532 + if earlier_images > 0 {

1533 + let key = (session_id.to_string(), model_name.to_string());

1534 + let mut noted = IMAGE_GAP_NOTED.lock().unwrap_or_else(|e| e.into_inner());

1535 + if !noted.contains(&key) {

1536 + noted.push(key);

1537 + let notice = format!(

1538 + "> **Earlier images are hidden.** {model_name} cannot read images, so the \

1539 + {earlier_images} {} attached earlier in this thread {} not sent to it. {HINT}",

1540 + plural(earlier_images),

1541 + if earlier_images == 1 { "is" } else { "are" }

1542 + );

1543 + return (user_text, images, Some(notice));

1544 + }

1545 + }

1546 + (user_text, images, None)

1547 +}

1548 +

1549 /// The servers out of a session request's `mcpServers` that siGit Code can

1550 /// connect to.

1551 ///

@@ -1577,6 +1647,11 @@ impl SiGitAgent {

1647 .agent_capabilities(

1648 AgentCapabilities::default()

1649 .load_session(true)

1650 + // Lets the editor attach images to a prompt. The capability is fixed

1651 + // for the connection while the model is not, so a prompt that

1652 + // brings an image to a model that cannot read one is answered with

1653 + // a note instead (see `images_for_turn`).

1654 + .prompt_capabilities(PromptCapabilities::new().image(true))

1655 // Clients only pass HTTP MCP servers in `mcpServers` to an

1656 // agent that says it can reach them. SSE stays off: the

1657 // MCP spec deprecated that transport.

@@ -2194,12 +2269,19 @@ impl SiGitAgent {

2269 }

2270

2271 let mut parts: Vec<String> = Vec::new();

2272 + let mut images: Vec<ImageInput> = Vec::new();

2273

2274 for block in &args.prompt {

2275 match block {

2276 ContentBlock::Text(t) => {

2277 parts.push(t.text.clone());

2278 }

2279 + ContentBlock::Image(image) => {

2280 + images.push(ImageInput {

2281 + mime_type: image.mime_type.clone(),

2282 + data: image.data.clone(),

2283 + });

2284 + }

2285 ContentBlock::Resource(embedded) => {

2286 // editor inlined the file content already

2287 match &embedded.resource {

@@ -2278,7 +2360,7 @@ impl SiGitAgent {

2360

2361 let user_text = parts.join("\n");

2362

2281 - if user_text.trim().is_empty() {

2363 + if user_text.trim().is_empty() && images.is_empty() {

2364 return Ok(PromptResponse::new(StopReason::EndTurn));

2365 }

2366

@@ -2362,6 +2444,34 @@ impl SiGitAgent {

2444

2445 let tools = agent_tools_as_specs();

2446

2447 + let model_name = self.current_model.lock().unwrap().display_name.clone();

2448 + let earlier_images = if backend.accepts_images() {

2449 + 0

2450 + } else {

2451 + backend

2452 + .history_snapshot()

2453 + .await

2454 + .iter()

2455 + .map(backend::message_image_count)

2456 + .sum()

2457 + };

2458 + let (user_text, images, notice) = images_for_turn(

2459 + user_text,

2460 + images,

2461 + backend.accepts_images(),

2462 + &model_name,

2463 + earlier_images,

2464 + &session_id.to_string(),

2465 + );

2466 + if let Some(notice) = notice {

2467 + self.send_assistant_message(

2468 + cx,

2469 + session_id.clone(),

2470 + format!("{notice}{PARAGRAPH_BREAK}"),

2471 + )

2472 + .ok();

2473 + }

2474 +

2475 // Token sink: backends stream assistant text through this while a turn

2476 // runs. We forward the visible portion to the editor as agent-message

2477 // chunks live (see `drain_turn` / `emit_visible_chunk`). The sink stays

@@ -2375,7 +2485,7 @@ impl SiGitAgent {

2485 .drain_turn(

2486 cx,

2487 &session_id,

2378 - backend.send_message_with_tools(&user_text, &tools, Some(&sink)),

2488 + backend.send_message_with_images(&user_text, &images, &tools, Some(&sink)),

2489 &mut sink_rx,

2490 &mut reply,

2491 &cancellation,

@@ -3494,6 +3604,16 @@ fn build_config_options(

3604 if item.tool_calling {

3605 desc_parts.push("tool calling".to_string());

3606 }

3607 + // The editor offers image attachments for every model, because

3608 + // the capability is per connection. This is where the user can see

3609 + // which models will actually look at one.

3610 + if item

3611 + .cloud_tier

3612 + .as_deref()

3613 + .is_some_and(provider::tier_accepts_images)

3614 + {

3615 + desc_parts.push("reads images".to_string());

3616 + }

3617 desc_parts.push(format!(

3618 "{} context",

3619 models::format_context_window_short(item.context_window_tokens)

@@ -5612,6 +5732,52 @@ mod tests {

5732 assert_eq!(longest_backtick_run("one ` two `` three ``` four"), 3);

5733 }

5734

5735 + fn png() -> ImageInput {

5736 + ImageInput {

5737 + mime_type: "image/png".to_string(),

5738 + data: "AAAA".to_string(),

5739 + }

5740 + }

5741 +

5742 + #[test]

5743 + fn images_go_through_untouched_to_a_model_that_reads_them() {

5744 + let (text, images, notice) =

5745 + images_for_turn("look".to_string(), vec![png()], true, "Large", 3, "s-pass");

5746 + assert_eq!(text, "look");

5747 + assert_eq!(images.len(), 1);

5748 + assert!(notice.is_none());

5749 + }

5750 +

5751 + #[test]

5752 + fn a_text_only_model_gets_a_note_and_the_user_is_told() {

5753 + let (text, images, notice) =

5754 + images_for_turn("look".to_string(), vec![png()], false, "Nova", 0, "s-drop");

5755 + assert!(images.is_empty());

5756 + assert!(

5757 + text.starts_with("look\n\n[The user attached 1 image."),

5758 + "{text}"

5759 + );

5760 + let notice = notice.expect("the user must be told the image was left out");

5761 + assert!(notice.contains("Nova cannot read images"), "{notice}");

5762 + assert!(notice.contains("reads images"), "{notice}");

5763 + }

5764 +

5765 + #[test]

5766 + fn switching_to_a_text_only_model_is_flagged_once() {

5767 + let turn = || images_for_turn("next".to_string(), Vec::new(), false, "Nova", 2, "s-switch");

5768 +

5769 + let (text, _, notice) = turn();

5770 + assert_eq!(text, "next", "nothing is added to a prompt with no image");

5771 + let notice = notice.expect("earlier images going out of view must be flagged");

5772 + assert!(notice.contains("2 images attached earlier"), "{notice}");

5773 +

5774 + let (_, _, again) = turn();

5775 + assert!(

5776 + again.is_none(),

5777 + "the note is shown once, not on every prompt"

5778 + );

5779 + }

5780 +

5781 #[test]

5782 fn session_roots_must_be_absolute() {

5783 let absolute = std::env::temp_dir();

src/provider.rs

+56

@@ -45,6 +45,40 @@ pub const CLOUD_TIERS: &[&str] = &[

46 pub const DEFAULT_CLOUD_TIER: &str = "nova";

48 +/// Cloud tiers that read images attached to a prompt.

49 +///

50 +/// This mirrors onde-cloud's `IMAGE_CANDIDATES` table, which is where the

51 +/// per-provider facts live and where a request is actually accepted or

52 +/// refused. A tier is listed here when at least one of its providers takes

53 +/// images. The rest are text-only at the source: `nova` and `orbit` (GLM-5.3

54 +/// and GLM-5.2), `apex` and `flux` (DeepSeek V4 Pro and V4 Flash).

55 +///

56 +/// The list can be hard-coded because the tier list itself is: a tier this

57 +/// client does not know is not offered at all (see [`CLOUD_TIERS`]).

58 +const IMAGE_TIERS: &[&str] = &[

59 + "fast", "balanced", "large", "mini", "air", "pro", "oke", "aura", "flare", "zenith", "prism",

60 +];

61 +

62 +/// Whether the cloud tier `tier` reads images.

63 +pub fn tier_accepts_images(tier: &str) -> bool {

64 + IMAGE_TIERS.contains(&tier.trim().to_lowercase().as_str())

65 +}

66 +

67 +/// Whether the model id sent on the wire reads images.

68 +///

69 +/// An `onde-*` id is one of our tiers and is answered from [`IMAGE_TIERS`]. Any

70 +/// other id belongs to an endpoint the user configured themselves

71 +/// (`OPENAI_BASE_URL` or a provider profile). Nothing is known about those, so

72 +/// the image is passed through and the endpoint gets to decide.

73 +pub fn model_accepts_images(model: &str) -> bool {

74 + if !model.starts_with("onde-") {

75 + return true;

76 + }

77 + CLOUD_TIERS

78 + .iter()

79 + .any(|tier| tier_to_model(tier) == model && tier_accepts_images(tier))

80 +}

81 +

82 /// Base URL of the siGit Code Cloud inference endpoint. Override with

83 /// `SIGIT_CLOUD_URL` (dev: `http://localhost:8090/v1`).

84 pub fn cloud_base_url() -> String {

@@ -282,4 +316,26 @@ mod tests {

316 assert_eq!(tier_to_model(tier), expected);

317 }

318 }

319 +

320 + #[test]

321 + fn image_support_follows_the_tier_table() {

322 + assert!(tier_accepts_images("large"));

323 + assert!(tier_accepts_images("oke"));

324 + // The default tier is text-only at the source.

325 + assert!(!tier_accepts_images(DEFAULT_CLOUD_TIER));

326 + assert!(!tier_accepts_images("apex"));

327 +

328 + assert!(model_accepts_images("onde-large"));

329 + // `oke` is the one tier whose wire id is not `onde-<tier>`.

330 + assert!(model_accepts_images("onde-kkk"));

331 + assert!(!model_accepts_images("onde-nova"));

332 + // An onde id this client has never heard of is not assumed to work.

333 + assert!(!model_accepts_images("onde-something-new"));

334 + // A model on the user's own endpoint is theirs to judge.

335 + assert!(model_accepts_images("gpt-4o-mini"));

336 +

337 + for tier in IMAGE_TIERS {

338 + assert!(CLOUD_TIERS.contains(tier), "{tier} is not a cloud tier");

339 + }

340 + }

341 }

tests/acp_images.rs new

+318

@@ -0,0 +1,318 @@

1 +//! End-to-end check of image attachments over ACP (issue #134).

2 +//!

3 +//! `promptCapabilities.image` is fixed for the connection, so the editor will

4 +//! attach an image whatever model is selected. Two things have to hold:

5 +//!

6 +//! - a model that reads images receives the image, as an OpenAI `image_url`

7 +//! content part carrying a base64 `data:` URL;

8 +//! - a model that does not is sent text only, with a note in place of the

9 +//! image, and the user is told the image was left out.

10 +//!

11 +//! Both run the real binary against a scripted OpenAI-compatible endpoint that

12 +//! records what it was sent. The model id decides which case applies: an

13 +//! `onde-*` id is looked up in the tier table, anything else is the user's own

14 +//! endpoint and is passed the image.

15 +

16 +use std::collections::VecDeque;

17 +use std::io::{BufRead, BufReader, Read, Write};

18 +use std::net::TcpListener;

19 +use std::process::{Child, ChildStdin, Command, Stdio};

20 +use std::sync::mpsc::{Receiver, channel};

21 +use std::sync::{Arc, Mutex};

22 +use std::time::{Duration, Instant};

23 +

24 +use serde_json::{Value, json};

25 +

26 +const TIMEOUT: Duration = Duration::from_secs(60);

27 +

28 +// ── Scripted OpenAI-compatible endpoint ─────────────────────────────────────

29 +

30 +fn sse_body(events: &[Value]) -> String {

31 + let mut body = String::new();

32 + for event in events {

33 + body.push_str("data: ");

34 + body.push_str(&event.to_string());

35 + body.push_str("\n\n");

36 + }

37 + body.push_str("data: [DONE]\n\n");

38 + body

39 +}

40 +

41 +fn sse_text(text: &str) -> String {

42 + sse_body(&[json!({"choices": [{"delta": {"content": text}}]})])

43 +}

44 +

45 +/// Serves one scripted SSE response per request and records each request body.

46 +struct FakeEndpoint {

47 + port: u16,

48 + requests: Arc<Mutex<Vec<Value>>>,

49 +}

50 +

51 +fn start_fake_endpoint(responses: Vec<String>) -> FakeEndpoint {

52 + let listener = TcpListener::bind("127.0.0.1:0").expect("bind fake endpoint");

53 + let port = listener.local_addr().unwrap().port();

54 + let requests: Arc<Mutex<Vec<Value>>> = Arc::default();

55 + let recorded = Arc::clone(&requests);

56 + let queue = Mutex::new(VecDeque::from(responses));

57 +

58 + std::thread::spawn(move || {

59 + for stream in listener.incoming() {

60 + let Ok(mut stream) = stream else { continue };

61 + let mut reader = BufReader::new(match stream.try_clone() {

62 + Ok(clone) => clone,

63 + Err(_) => continue,

64 + });

65 + let mut content_length = 0usize;

66 + loop {

67 + let mut line = String::new();

68 + if reader.read_line(&mut line).unwrap_or(0) == 0 {

69 + break;

70 + }

71 + let line = line.trim();

72 + if line.is_empty() {

73 + break;

74 + }

75 + if let Some(length) = line.to_ascii_lowercase().strip_prefix("content-length:") {

76 + content_length = length.trim().parse().unwrap_or(0);

77 + }

78 + }

79 + let mut body = vec![0u8; content_length];

80 + if reader.read_exact(&mut body).is_err() {

81 + continue;

82 + }

83 + if let Ok(request) = serde_json::from_slice::<Value>(&body) {

84 + recorded.lock().unwrap().push(request);

85 + }

86 + let payload = queue

87 + .lock()

88 + .unwrap()

89 + .pop_front()

90 + .unwrap_or_else(|| sse_text("out of scripted responses"));

91 + let response = format!(

92 + "HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\n\

93 + content-length: {}\r\nconnection: close\r\n\r\n{}",

94 + payload.len(),

95 + payload

96 + );

97 + let _ = stream.write_all(response.as_bytes());

98 + }

99 + });

100 +

101 + FakeEndpoint { port, requests }

102 +}

103 +

104 +// ── ACP client over the binary's stdio ──────────────────────────────────────

105 +

106 +struct AgentUnderTest {

107 + child: Child,

108 + stdin: ChildStdin,

109 + incoming: Receiver<Value>,

110 + next_id: u64,

111 +}

112 +

113 +fn spawn_agent(port: u16, config_dir: &std::path::Path, model: &str) -> AgentUnderTest {

114 + let mut child = Command::new(env!("CARGO_BIN_EXE_sigit"))

115 + .env("OPENAI_BASE_URL", format!("http://127.0.0.1:{port}"))

116 + .env("OPENAI_API_KEY", "test-key")

117 + .env("SIGIT_MODEL", model)

118 + .env("SIGIT_CONFIG_DIR", config_dir)

119 + .env("SIGIT_MCP", "off")

120 + .env("SIGIT_PERMISSIONS", "allow")

121 + .env_remove("SIGIT_LOCAL_INFERENCE")

122 + .stdin(Stdio::piped())

123 + .stdout(Stdio::piped())

124 + .stderr(Stdio::null())

125 + .spawn()

126 + .expect("spawn sigit in ACP mode");

127 +

128 + let stdout = child.stdout.take().unwrap();

129 + let (message_tx, incoming) = channel();

130 + std::thread::spawn(move || {

131 + for line in BufReader::new(stdout).lines() {

132 + let Ok(line) = line else { break };

133 + if let Ok(message) = serde_json::from_str::<Value>(&line)

134 + && message_tx.send(message).is_err()

135 + {

136 + break;

137 + }

138 + }

139 + });

140 +

141 + let stdin = child.stdin.take().unwrap();

142 + AgentUnderTest {

143 + child,

144 + stdin,

145 + incoming,

146 + next_id: 0,

147 + }

148 +}

149 +

150 +impl AgentUnderTest {

151 + fn send(&mut self, message: Value) {

152 + let mut line = message.to_string();

153 + line.push('\n');

154 + self.stdin

155 + .write_all(line.as_bytes())

156 + .expect("write to agent stdin");

157 + self.stdin.flush().expect("flush agent stdin");

158 + }

159 +

160 + fn request(&mut self, method: &str, params: Value) -> u64 {

161 + self.next_id += 1;

162 + let id = self.next_id;

163 + self.send(json!({"jsonrpc": "2.0", "id": id, "method": method, "params": params}));

164 + id

165 + }

166 +

167 + /// Wait for the response to our request `id`, collecting the raw JSON of

168 + /// every `session/update` notification that arrives before it.

169 + fn wait_for_response_collecting_updates(&mut self, id: u64) -> (Value, String) {

170 + let deadline = Instant::now() + TIMEOUT;

171 + let mut updates = String::new();

172 + loop {

173 + let remaining = deadline.saturating_duration_since(Instant::now());

174 + match self.incoming.recv_timeout(remaining) {

175 + Ok(message) if message["id"] == id && message.get("method").is_none() => {

176 + assert!(

177 + message.get("error").is_none(),

178 + "request {id} failed: {message}"

179 + );

180 + return (message, updates);

181 + }

182 + Ok(message) => {

183 + if message["method"] == "session/update" {

184 + updates.push_str(&message["params"].to_string());

185 + updates.push('\n');

186 + }

187 + }

188 + Err(_) => panic!("timed out waiting for response to request {id}"),

189 + }

190 + }

191 + }

192 +

193 + fn wait_for_response(&mut self, id: u64) -> Value {

194 + self.wait_for_response_collecting_updates(id).0

195 + }

196 +}

197 +

198 +impl Drop for AgentUnderTest {

199 + fn drop(&mut self) {

200 + let _ = self.child.kill();

201 + let _ = self.child.wait();

202 + }

203 +}

204 +

205 +// ── The two cases ────────────────────────────────────────────────────────────

206 +

207 +/// A 1x1 PNG, base64-encoded.

208 +const PIXEL: &str = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==";

209 +

210 +/// Run one prompt that carries text and an image against `model`. Returns the

211 +/// `initialize` response, the raw `session/update` stream of the prompt, and

212 +/// the first completion request the endpoint received.

213 +fn prompt_with_image(model: &str, tag: &str) -> (Value, String, Value) {

214 + let endpoint = start_fake_endpoint(vec![sse_text("done")]);

215 + let scratch =

216 + std::env::temp_dir().join(format!("sigit_acp_images_{tag}_{}", std::process::id()));

217 + let config_dir = scratch.join("config");

218 + let project = scratch.join("project");

219 + std::fs::create_dir_all(&config_dir).unwrap();

220 + std::fs::create_dir_all(&project).unwrap();

221 +

222 + let mut agent = spawn_agent(endpoint.port, &config_dir, model);

223 +

224 + let id = agent.request(

225 + "initialize",

226 + json!({"protocolVersion": 1, "clientCapabilities": {}}),

227 + );

228 + let initialize = agent.wait_for_response(id);

229 +

230 + let id = agent.request("session/new", json!({"cwd": project, "mcpServers": []}));

231 + let session_id = agent.wait_for_response(id)["result"]["sessionId"]

232 + .as_str()

233 + .expect("session id")

234 + .to_string();

235 +

236 + let id = agent.request(

237 + "session/prompt",

238 + json!({

239 + "sessionId": session_id,

240 + "prompt": [

241 + {"type": "text", "text": "what is in this picture?"},

242 + {"type": "image", "mimeType": "image/png", "data": PIXEL},

243 + ],

244 + }),

245 + );

246 + let (response, updates) = agent.wait_for_response_collecting_updates(id);

247 + assert_eq!(response["result"]["stopReason"], "end_turn");

248 +

249 + let request = endpoint

250 + .requests

251 + .lock()

252 + .unwrap()

253 + .first()

254 + .cloned()

255 + .expect("the endpoint received a completion request");

256 +

257 + drop(agent);

258 + let _ = std::fs::remove_dir_all(&scratch);

259 + (initialize, updates, request)

260 +}

261 +

262 +fn last_user_message(request: &Value) -> &Value {

263 + request["messages"]

264 + .as_array()

265 + .expect("messages")

266 + .iter()

267 + .rev()

268 + .find(|message| message["role"] == "user")

269 + .expect("a user message")

270 +}

271 +

272 +#[test]

273 +fn an_image_reaches_a_model_that_reads_images() {

274 + let (initialize, updates, request) = prompt_with_image("scripted-vision-model", "vision");

275 +

276 + assert_eq!(

277 + initialize["result"]["agentCapabilities"]["promptCapabilities"]["image"], true,

278 + "a client only attaches images for an agent that advertises them: {initialize}"

279 + );

280 +

281 + let message = last_user_message(&request);

282 + assert_eq!(

283 + message["content"],

284 + json!([

285 + {"type": "text", "text": "what is in this picture?"},

286 + {"type": "image_url", "image_url": {"url": format!("data:image/png;base64,{PIXEL}")}},

287 + ]),

288 + "the image must travel as an image_url content part"

289 + );

290 + assert!(

291 + !updates.contains("Image not sent"),

292 + "nothing was left out, so nothing should be flagged: {updates}"

293 + );

294 +}

295 +

296 +#[test]

297 +fn a_text_only_model_gets_the_text_and_the_user_is_told() {

298 + // `onde-nova` is the default cloud tier and is text-only at the source.

299 + let (_, updates, request) = prompt_with_image("onde-nova", "text_only");

300 +

301 + let message = last_user_message(&request);

302 + let content = message["content"]

303 + .as_str()

304 + .unwrap_or_else(|| panic!("a text-only model must get string content: {message}"));

305 + assert!(content.starts_with("what is in this picture?"), "{content}");

306 + assert!(

307 + content.contains("This model cannot read images"),

308 + "the model must know an image was withheld: {content}"

309 + );

310 + assert!(

311 + !request.to_string().contains(PIXEL),

312 + "no image data may be sent to a model that cannot read it"

313 + );

314 + assert!(

315 + updates.contains("Image not sent"),

316 + "the user must be told the image was left out: {updates}"

317 + );

318 +}