| 1 | # Skill: AI-Assisted Coding Agents — Onde Inference Integration |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | Building a local AI coding agent in Rust using Onde Inference as the LLM backend. |
| 6 | Onde wraps mistral.rs with a clean API for model loading, history management, and |
| 7 | streaming inference across macOS (Metal), iOS, Android, Linux, and Windows. |
| 8 | |
| 9 | Crate: `onde = { path = "../onde" }` or from crates.io when published |
| 10 | Repo: https://github.com/ondeinference/onde |
| 11 | Docs: https://ondeinference.com |
| 12 | |
| 13 | --- |
| 14 | |
| 15 | ## Onde `ChatEngine` API |
| 16 | |
| 17 | ### Construction and lifecycle |
| 18 | |
| 19 | ```rust |
| 20 | use onde::inference::{ChatEngine, GgufModelConfig, SamplingConfig}; |
| 21 | |
| 22 | let engine = ChatEngine::new(); // starts unloaded |
| 23 | engine.is_loaded().await // -> bool |
| 24 | engine.unload_model().await // -> () |
| 25 | ``` |
| 26 | |
| 27 | ### Loading a model |
| 28 | |
| 29 | ```rust |
| 30 | // Platform-aware default (Qwen 2.5 3B on macOS, 1.5B on iOS/tvOS/Android) |
| 31 | let config = GgufModelConfig::platform_default(); |
| 32 | |
| 33 | // Load — blocks until model is in memory and on GPU |
| 34 | engine |
| 35 | .load_gguf_model( |
| 36 | config, |
| 37 | Some("You are a helpful assistant.".to_string()), // system prompt |
| 38 | None, // sampling config (uses SamplingConfig::default() internally) |
| 39 | ) |
| 40 | .await?; |
| 41 | |
| 42 | // AlreadyLoaded error if called twice — check first: |
| 43 | if !engine.is_loaded().await { |
| 44 | engine.load_gguf_model(...).await?; |
| 45 | } |
| 46 | ``` |
| 47 | |
| 48 | **Model sizes (macOS/Windows/Linux default — Qwen 2.5 3B Q4_K_M):** ~1.93 GB |
| 49 | **Model sizes (iOS/tvOS/Android default — Qwen 2.5 1.5B Q4_K_M):** ~941 MB |
| 50 | First run downloads from HuggingFace Hub into `~/.cache/huggingface/`. |
| 51 | |
| 52 | ### Blocking (non-streaming) inference |
| 53 | |
| 54 | ```rust |
| 55 | let result = engine.send_message("What is Rust's ownership model?").await?; |
| 56 | // result: InferenceResult |
| 57 | println!("{}", result.text); |
| 58 | println!("took {}", result.duration_display); // e.g. "3.2s" |
| 59 | ``` |
| 60 | |
| 61 | `send_message` appends both the user message and assistant reply to conversation |
| 62 | history automatically. |
| 63 | |
| 64 | ### Streaming inference |
| 65 | |
| 66 | ```rust |
| 67 | let mut rx: tokio::sync::mpsc::Receiver<StreamChunk> = |
| 68 | engine.stream_message("Tell me a story.").await?; |
| 69 | |
| 70 | while let Some(chunk) = rx.recv().await { |
| 71 | if !chunk.delta.is_empty() { |
| 72 | print!("{}", chunk.delta); // partial token text |
| 73 | } |
| 74 | if chunk.done { |
| 75 | // chunk.finish_reason: Option<String> — e.g. "stop", "length" |
| 76 | break; |
| 77 | } |
| 78 | } |
| 79 | ``` |
| 80 | |
| 81 | `StreamChunk` fields: |
| 82 | - `delta: String` — the new token(s) in this chunk |
| 83 | - `done: bool` — true on the last chunk |
| 84 | - `finish_reason: Option<String>` — present on final chunk only |
| 85 | |
| 86 | History is updated automatically after the stream completes. |
| 87 | |
| 88 | ### One-shot generation (no history side-effects) |
| 89 | |
| 90 | ```rust |
| 91 | use onde::inference::ChatMessage; |
| 92 | |
| 93 | let result = engine.generate( |
| 94 | vec![ChatMessage::user("Expand: a cat in space")], |
| 95 | Some(SamplingConfig::deterministic()), |
| 96 | ).await?; |
| 97 | println!("{}", result.text); |
| 98 | // Does NOT modify conversation history |
| 99 | ``` |
| 100 | |
| 101 | ### History management |
| 102 | |
| 103 | ```rust |
| 104 | let history: Vec<ChatMessage> = engine.history().await; |
| 105 | let removed: usize = engine.clear_history().await; // returns count cleared |
| 106 | engine.push_history(ChatMessage::user("context")).await; |
| 107 | engine.set_system_prompt("new system prompt").await; |
| 108 | engine.clear_system_prompt().await; |
| 109 | ``` |
| 110 | |
| 111 | ### Engine status |
| 112 | |
| 113 | ```rust |
| 114 | let info: EngineInfo = engine.info().await; |
| 115 | // info.status: EngineStatus (Unloaded | Loading | Ready | Generating | Error) |
| 116 | // info.model_name: Option<String> |
| 117 | // info.approx_memory: Option<String> e.g. "~1.93 GB" |
| 118 | // info.history_length: u64 |
| 119 | ``` |
| 120 | |
| 121 | --- |
| 122 | |
| 123 | ## `InferenceError` variants |
| 124 | |
| 125 | ```rust |
| 126 | match err { |
| 127 | InferenceError::NoModelLoaded => { /* load model first */ } |
| 128 | InferenceError::AlreadyLoaded { model_name } => { /* already loaded */ } |
| 129 | InferenceError::ModelBuild { reason } => { /* load failure */ } |
| 130 | InferenceError::Inference { reason } => { /* runtime inference error */ } |
| 131 | InferenceError::Cancelled => { /* was cancelled */ } |
| 132 | InferenceError::Other { reason } => { /* unexpected */ } |
| 133 | } |
| 134 | ``` |
| 135 | |
| 136 | Map to ACP errors: |
| 137 | ```rust |
| 138 | .map_err(|e| agent_client_protocol::Error::new(-32603, e.to_string()))? |
| 139 | ``` |
| 140 | |
| 141 | --- |
| 142 | |
| 143 | ## `SamplingConfig` presets |
| 144 | |
| 145 | | Preset | temp | top_p | max_tokens | Use case | |
| 146 | |--------|------|-------|------------|----------| |
| 147 | | `SamplingConfig::default()` | 0.7 | 0.95 | 512 | General chat | |
| 148 | | `SamplingConfig::deterministic()` | 0.0 | — | 512 | Code / reproducible | |
| 149 | | `SamplingConfig::mobile()` | 0.7 | 0.95 | 128 | Memory-constrained | |
| 150 | | `SamplingConfig::coding()` | 0.0 | — | 512 | Code generation | |
| 151 | | `SamplingConfig::coding_mobile()` | 0.0 | — | 128 | Code on mobile | |
| 152 | |
| 153 | --- |
| 154 | |
| 155 | ## `GgufModelConfig` constructors |
| 156 | |
| 157 | ```rust |
| 158 | GgufModelConfig::platform_default() // auto-selects based on target_os |
| 159 | GgufModelConfig::qwen25_1_5b() // force 1.5B |
| 160 | GgufModelConfig::qwen25_3b() // force 3B |
| 161 | GgufModelConfig::qwen25_coder_1_5b() // coder variant 1.5B |
| 162 | GgufModelConfig::qwen25_coder_3b() // coder variant 3B |
| 163 | ``` |
| 164 | |
| 165 | --- |
| 166 | |
| 167 | ## Adding onde as a Rust library dependency |
| 168 | |
| 169 | ```toml |
| 170 | # In your crate's Cargo.toml — onde is a path dep since it's not on crates.io yet |
| 171 | onde = { path = "../onde" } |
| 172 | ``` |
| 173 | |
| 174 | **Important:** `onde` declares `crate-type = ["lib", "cdylib", "staticlib"]`. |
| 175 | When used as a Rust library dep, only the `lib` target is compiled. The |
| 176 | `cdylib`/`staticlib` targets (used for Swift/Kotlin FFI) are not built. The |
| 177 | `uniffi::setup_scaffolding!()` macro generates `#[no_mangle] extern "C"` symbols |
| 178 | but these are harmless in a binary context. |
| 179 | |
| 180 | **The `[patch.crates-io]` in onde's Cargo.toml does NOT propagate** to dependents |
| 181 | unless they are in the same workspace. The `sysctl` patch is only needed for |
| 182 | watchOS; macOS/iOS/Linux work without it. |
| 183 | |
| 184 | **GPU feature selection is automatic** via `target_os` cfg flags in onde's |
| 185 | Cargo.toml — you get Metal on macOS/iOS without any extra features in your crate. |
| 186 | |
| 187 | --- |
| 188 | |
| 189 | ## Patterns for coding agents |
| 190 | |
| 191 | ### Single-engine, multi-session via history reset |
| 192 | |
| 193 | For a simple MVP where one session is active at a time: |
| 194 | |
| 195 | ```rust |
| 196 | struct MyAgent { |
| 197 | engine: Arc<ChatEngine>, |
| 198 | active_session: Arc<Mutex<Option<SessionId>>>, |
| 199 | } |
| 200 | |
| 201 | // new_session handler: |
| 202 | if self.engine.is_loaded().await { |
| 203 | self.engine.clear_history().await; // reuse model, fresh conversation |
| 204 | } else { |
| 205 | self.engine |
| 206 | .load_gguf_model(GgufModelConfig::platform_default(), Some(SYSTEM_PROMPT.into()), None) |
| 207 | .await?; |
| 208 | } |
| 209 | ``` |
| 210 | |
| 211 | **Why:** Loading the model is expensive (seconds + GB of RAM). Reloading for each |
| 212 | session would make the agent feel broken. `clear_history()` resets context in |
| 213 | microseconds. |
| 214 | |
| 215 | ### Per-session engines (multiple concurrent sessions) |
| 216 | |
| 217 | When you need truly isolated parallel sessions: |
| 218 | |
| 219 | ```rust |
| 220 | use std::collections::HashMap; |
| 221 | |
| 222 | struct MultiSessionAgent { |
| 223 | sessions: Arc<Mutex<HashMap<String, Arc<ChatEngine>>>>, |
| 224 | } |
| 225 | |
| 226 | // new_session: create and load a new engine per session |
| 227 | // prompt: look up session engine, call send_message or stream_message |
| 228 | // CAVEAT: each engine holds a separate model copy in GPU memory — expensive! |
| 229 | ``` |
| 230 | |
| 231 | Better approach for shared GPU memory: use `engine.generate()` (no history |
| 232 | side-effects) with an explicitly managed message vec per session. |
| 233 | |
| 234 | ### System prompt design for coding agents |
| 235 | |
| 236 | ```rust |
| 237 | const SYSTEM_PROMPT: &str = "\ |
| 238 | You are <AgentName>, an expert AI coding agent integrated into your editor \ |
| 239 | via the Agent Client Protocol. You specialize in: |
| 240 | |
| 241 | - Code analysis, writing, and refactoring |
| 242 | - Bug hunting and debugging |
| 243 | - Git workflows and commit messages |
| 244 | - Software architecture and design patterns |
| 245 | - Code review and best practices |
| 246 | |
| 247 | Be concise, precise, and practical. Write clean, idiomatic code with brief \ |
| 248 | explanations. Identify root causes when debugging. Prefer correctness over brevity."; |
| 249 | ``` |
| 250 | |
| 251 | Key principles: |
| 252 | - State the agent's role and name clearly (models respond better to named personas) |
| 253 | - List specializations explicitly (influences which parts of training are activated) |
| 254 | - Set tone expectations: "concise", "practical", "idiomatic" |
| 255 | - Avoid verbose instruction lists — they cost tokens on every turn |
| 256 | |
| 257 | ### Streaming tokens to ACP (connecting onde → ACP) |
| 258 | |
| 259 | ```rust |
| 260 | // In Agent::prompt(): |
| 261 | let mut rx = self.engine.stream_message(user_text).await |
| 262 | .map_err(|e| Error::new(-32603, e.to_string()))?; |
| 263 | |
| 264 | while let Some(chunk) = rx.recv().await { |
| 265 | if !chunk.delta.is_empty() { |
| 266 | self.notification_tx.send( |
| 267 | SessionNotification::new( |
| 268 | session_id.clone(), |
| 269 | SessionUpdate::AgentMessageChunk( |
| 270 | ContentChunk::new(ContentBlock::from(chunk.delta)), |
| 271 | ), |
| 272 | ) |
| 273 | ).await.ok(); // .ok() — ignore if forwarder is gone |
| 274 | } |
| 275 | if chunk.done { break; } |
| 276 | } |
| 277 | |
| 278 | Ok(PromptResponse::new(StopReason::EndTurn)) |
| 279 | ``` |
| 280 | |
| 281 | The `PromptResponse` is returned AFTER the stream finishes. The client receives |
| 282 | streaming tokens via `session/update` notifications while blocking on the |
| 283 | `session/prompt` response. |
| 284 | |
| 285 | --- |
| 286 | |
| 287 | ## Extracting text from ACP `PromptRequest` |
| 288 | |
| 289 | ACP prompts can contain text, images, resource links, etc. For a text-only |
| 290 | coding agent: |
| 291 | |
| 292 | ```rust |
| 293 | let user_text: String = args.prompt.iter() |
| 294 | .filter_map(|block| match block { |
| 295 | ContentBlock::Text(t) => Some(t.text.as_str()), |
| 296 | // Skip images, resource links, embedded resources for now |
| 297 | _ => None, |
| 298 | }) |
| 299 | .collect::<Vec<_>>() |
| 300 | .join("\n"); |
| 301 | ``` |
| 302 | |
| 303 | For future resource context (e.g. open files provided by Zed): |
| 304 | ```rust |
| 305 | ContentBlock::Resource(r) => match &r.resource { |
| 306 | EmbeddedResourceResource::Text(t) => Some(t.text.as_str()), |
| 307 | _ => None, |
| 308 | }, |
| 309 | ``` |
| 310 | |
| 311 | --- |
| 312 | |
| 313 | ## `ChatEngine` threading model |
| 314 | |
| 315 | - Internally uses `Arc<tokio::sync::Mutex<Option<LoadedModel>>>` — `Send + Sync`. |
| 316 | - Safe to wrap in `Arc<ChatEngine>` and share across tasks. |
| 317 | - `stream_message()` spawns a `tokio::spawn` background task internally — the |
| 318 | mistralrs model must be `Send`, which it is on all supported platforms. |
| 319 | - Calling `stream_message()` from a `!Send` future (e.g. inside a `LocalSet`) is |
| 320 | fine — the future itself doesn't hold a `!Send` value across `.await`. |
| 321 | |
| 322 | --- |
| 323 | |
| 324 | ## First-run model download |
| 325 | |
| 326 | On first use, onde downloads the GGUF model from HuggingFace Hub: |
| 327 | - Requires internet connectivity |
| 328 | - Cached at `~/.cache/huggingface/` (or `HF_HUB_CACHE` env var) |
| 329 | - `HF_TOKEN` env var needed for gated models (public Qwen models don't need it) |
| 330 | - Subsequent runs load from disk cache — fast |
| 331 | |
| 332 | For sandboxed environments (iOS, tvOS, Android): |
| 333 | - Set `HF_HOME` and `HF_HUB_CACHE` to a path inside the app container |
| 334 | - Do this BEFORE calling any ChatEngine method |
| 335 | - See `onde/docs/swift-package.md` for `setupInferenceEnvironment()` pattern |
| 336 | |
| 337 | --- |
| 338 | |
| 339 | ## Common mistakes |
| 340 | |
| 341 | 1. **Calling `load_gguf_model` twice** without checking `is_loaded()` first → |
| 342 | `InferenceError::AlreadyLoaded`. Always guard with `is_loaded().await`. |
| 343 | |
| 344 | 2. **Blocking on the stream after the channel is closed** → the stream naturally |
| 345 | ends when the `done` flag is true. Don't `recv()` after `done`. |
| 346 | |
| 347 | 3. **Losing `StreamChunk` deltas** when `delta` is empty (whitespace tokens) → |
| 348 | always check `!chunk.delta.is_empty()` before sending to avoid empty |
| 349 | notifications that waste bandwidth. |
| 350 | |
| 351 | 4. **Sharing one `ChatEngine` across parallel prompts** without coordination → |
| 352 | the internal Mutex serializes inference, so concurrent prompts queue up. |
| 353 | Design for sequential access per engine instance. |
| 354 | |
| 355 | 5. **Using `SamplingConfig::default()` for code generation** → prefer |
| 356 | `SamplingConfig::coding()` (deterministic, temp=0) for more reliable code output. |
| 357 | |
| 358 | 6. **Forgetting that `generate()` doesn't update history** — use it for |
| 359 | one-shot enhancements (prompt expansion, code review) that shouldn't pollute |
| 360 | the main conversation. Use `send_message()` / `stream_message()` for the |
| 361 | primary turn loop. |