@setoelkahfi / sigit
Code Commits Pulls CI/CD 248
development
claude/code-feature-parity-q003hm claude/elegant-carson-l1menh claude/sigit-acp-local-chat-cx6380 claude/sigit-cloud-agent-expansion-reox0a claude/tool-permission-system claude/zen-feynman-0u78dk development feature/agent-tools-multiedit-glob-todos-remember feature/background-commands feature/commit-coauthor-attribution feature/headless-mode feature/init-command feature/load-local-model-explicitly feature/session-persistence-compaction feature/sigit-code-cloud feature/subagent-tool feature/tool-permission-system feature/tui-repo-tabs feature/tui-tabs main release/v1.3.1
md 390 lines 13.4 KB
Rendered Raw
1 ---
2 name: ai-assisted-coding
3 description: Build or maintain AI-assisted coding features in Rust using Onde Inference. Use when working on ChatEngine integration, model loading, streaming inference, history management, sampling config, or local coding-agent architecture.
4 ---
5
6 # Skill: AI-Assisted Coding Agents — Onde Inference Integration
7
8 ## Overview
9
10 Building a local AI coding agent in Rust using Onde Inference as the LLM backend.
11 Onde wraps mistral.rs with a clean API for model loading, history management, and
12 streaming inference across macOS (Metal), iOS, Android, Linux, and Windows.
13
14 Crate: `onde = "1.1.2"` (published on crates.io; siGit pins it in `Cargo.toml`)
15 Repo: https://github.com/ondeinference/onde
16 Docs: https://ondeinference.com
17
18 ---
19
20 ## Onde `ChatEngine` API
21
22 ### Construction and lifecycle
23
24 ```rust
25 use onde::inference::{ChatEngine, GgufModelConfig, SamplingConfig};
26
27 let engine = ChatEngine::new(); // starts unloaded
28 engine.is_loaded().await // -> bool
29 engine.unload_model().await // -> ()
30 ```
31
32 ### Loading a model
33
34 ```rust
35 // Platform-aware default (Qwen 2.5 3B on macOS, 1.5B on iOS/tvOS/Android)
36 let config = GgufModelConfig::platform_default();
37
38 // Load — blocks until model is in memory and on GPU
39 engine
40 .load_gguf_model(
41 config,
42 Some("You are a helpful assistant.".to_string()), // system prompt
43 None, // sampling config (uses SamplingConfig::default() internally)
44 )
45 .await?;
46
47 // AlreadyLoaded error if called twice — check first:
48 if !engine.is_loaded().await {
49 engine.load_gguf_model(...).await?;
50 }
51 ```
52
53 **Model sizes (macOS/Windows/Linux default — Qwen 2.5 3B Q4_K_M):** ~1.93 GB
54 **Model sizes (iOS/tvOS/Android default — Qwen 2.5 1.5B Q4_K_M):** ~941 MB
55 First run downloads from HuggingFace Hub into `~/.cache/huggingface/`.
56
57 ### Blocking (non-streaming) inference
58
59 ```rust
60 let result = engine.send_message("What is Rust's ownership model?").await?;
61 // result: InferenceResult
62 println!("{}", result.text);
63 println!("took {}", result.duration_display); // e.g. "3.2s"
64 ```
65
66 `send_message` appends both the user message and assistant reply to conversation
67 history automatically.
68
69 ### Streaming inference
70
71 ```rust
72 let mut rx: tokio::sync::mpsc::Receiver<StreamChunk> =
73 engine.stream_message("Tell me a story.").await?;
74
75 while let Some(chunk) = rx.recv().await {
76 if !chunk.delta.is_empty() {
77 print!("{}", chunk.delta); // partial token text
78 }
79 if chunk.done {
80 // chunk.finish_reason: Option<String> — e.g. "stop", "length"
81 break;
82 }
83 }
84 ```
85
86 `StreamChunk` fields:
87 - `delta: String` — the new token(s) in this chunk
88 - `done: bool` — true on the last chunk
89 - `finish_reason: Option<String>` — present on final chunk only
90
91 History is updated automatically after the stream completes.
92
93 ### One-shot generation (no history side-effects)
94
95 ```rust
96 use onde::inference::ChatMessage;
97
98 let result = engine.generate(
99 vec![ChatMessage::user("Expand: a cat in space")],
100 Some(SamplingConfig::deterministic()),
101 ).await?;
102 println!("{}", result.text);
103 // Does NOT modify conversation history
104 ```
105
106 ### History management
107
108 ```rust
109 let history: Vec<ChatMessage> = engine.history().await;
110 let removed: usize = engine.clear_history().await; // returns count cleared
111 engine.push_history(ChatMessage::user("context")).await;
112 engine.set_system_prompt("new system prompt").await;
113 engine.clear_system_prompt().await;
114 ```
115
116 ### Engine status
117
118 ```rust
119 let info: EngineInfo = engine.info().await;
120 // info.status: EngineStatus (Unloaded | Loading | Ready | Generating | Error)
121 // info.model_name: Option<String>
122 // info.approx_memory: Option<String> e.g. "~1.93 GB"
123 // info.history_length: u64
124 ```
125
126 ---
127
128 ## `InferenceError` variants
129
130 ```rust
131 match err {
132 InferenceError::NoModelLoaded => { /* load model first */ }
133 InferenceError::AlreadyLoaded { model_name } => { /* already loaded */ }
134 InferenceError::ModelBuild { reason } => { /* load failure */ }
135 InferenceError::Inference { reason } => { /* runtime inference error */ }
136 InferenceError::Cancelled => { /* was cancelled */ }
137 InferenceError::Other { reason } => { /* unexpected */ }
138 }
139 ```
140
141 Map to ACP errors:
142 ```rust
143 .map_err(|e| agent_client_protocol::Error::new(-32603, e.to_string()))?
144 ```
145
146 ---
147
148 ## `SamplingConfig` presets
149
150 | Preset | temp | top_p | max_tokens | Use case |
151 |--------|------|-------|------------|----------|
152 | `SamplingConfig::default()` | 0.7 | 0.95 | 512 | General chat |
153 | `SamplingConfig::deterministic()` | 0.0 | — | 512 | Code / reproducible |
154 | `SamplingConfig::mobile()` | 0.7 | 0.95 | 128 | Memory-constrained |
155 | `SamplingConfig::coding()` | 0.0 | — | 512 | Code generation |
156 | `SamplingConfig::coding_mobile()` | 0.0 | — | 128 | Code on mobile |
157
158 ---
159
160 ## `GgufModelConfig` constructors
161
162 ```rust
163 GgufModelConfig::platform_default() // auto-selects based on target_os
164 GgufModelConfig::qwen25_1_5b() // force 1.5B
165 GgufModelConfig::qwen25_3b() // force 3B
166 GgufModelConfig::qwen25_coder_1_5b() // coder variant 1.5B
167 GgufModelConfig::qwen25_coder_3b() // coder variant 3B
168 GgufModelConfig::qwen25_coder_7b() // coder variant 7B (tool calling)
169 GgufModelConfig::qwen3_1_7b() // Qwen 3 1.7B (tool calling)
170 GgufModelConfig::qwen3_4b() // Qwen 3 4B (tool calling)
171 GgufModelConfig::qwen3_8b() // Qwen 3 8B (tool calling)
172 GgufModelConfig::qwen3_14b() // Qwen 3 14B (tool calling)
173 ```
174
175 Only the Qwen 3 family and Qwen 2.5 Coder 7B support tool calling — see the
176 `tool-calling` skill. The on-device default is the saved selection, falling back
177 to `platform_default()` (Qwen 2.5 3B on macOS).
178
179 ---
180
181 ## Adding onde as a Rust library dependency
182
183 ```toml
184 # In your crate's Cargo.toml — onde is published on crates.io
185 onde = "1.1.2"
186 # For local SDK development against a checkout, swap to a path dep:
187 # onde = { path = "../onde" }
188 ```
189
190 **Important:** `onde` declares `crate-type = ["lib", "cdylib", "staticlib"]`.
191 When used as a Rust library dep, only the `lib` target is compiled. The
192 `cdylib`/`staticlib` targets (used for Swift/Kotlin FFI) are not built. The
193 `uniffi::setup_scaffolding!()` macro generates `#[no_mangle] extern "C"` symbols
194 but these are harmless in a binary context.
195
196 **The `[patch.crates-io]` in onde's Cargo.toml does NOT propagate** to dependents
197 unless they are in the same workspace. The `sysctl` patch is only needed for
198 watchOS; macOS/iOS/Linux work without it.
199
200 **GPU feature selection is automatic** via `target_os` cfg flags in onde's
201 Cargo.toml — you get Metal on macOS/iOS without any extra features in your crate.
202
203 ---
204
205 ## Patterns for coding agents
206
207 ### Single-engine, multi-session via history reset
208
209 For a simple MVP where one session is active at a time:
210
211 ```rust
212 struct MyAgent {
213 engine: Arc<ChatEngine>,
214 active_session: Arc<Mutex<Option<SessionId>>>,
215 }
216
217 // new_session handler:
218 if self.engine.is_loaded().await {
219 self.engine.clear_history().await; // reuse model, fresh conversation
220 } else {
221 self.engine
222 .load_gguf_model(GgufModelConfig::platform_default(), Some(SYSTEM_PROMPT.into()), None)
223 .await?;
224 }
225 ```
226
227 **Why:** Loading the model is expensive (seconds + GB of RAM). Reloading for each
228 session would make the agent feel broken. `clear_history()` resets context in
229 microseconds.
230
231 ### Per-session engines (multiple concurrent sessions)
232
233 When you need truly isolated parallel sessions:
234
235 ```rust
236 use std::collections::HashMap;
237
238 struct MultiSessionAgent {
239 sessions: Arc<Mutex<HashMap<String, Arc<ChatEngine>>>>,
240 }
241
242 // new_session: create and load a new engine per session
243 // prompt: look up session engine, call send_message or stream_message
244 // CAVEAT: each engine holds a separate model copy in GPU memory — expensive!
245 ```
246
247 Better approach for shared GPU memory: use `engine.generate()` (no history
248 side-effects) with an explicitly managed message vec per session.
249
250 ### System prompt design for coding agents
251
252 ```rust
253 const SYSTEM_PROMPT: &str = "\
254 You are <AgentName>, an expert AI coding agent integrated into your editor \
255 via the Agent Client Protocol. You specialize in:
256
257 - Code analysis, writing, and refactoring
258 - Bug hunting and debugging
259 - Git workflows and commit messages
260 - Software architecture and design patterns
261 - Code review and best practices
262
263 Be concise, precise, and practical. Write clean, idiomatic code with brief \
264 explanations. Identify root causes when debugging. Prefer correctness over brevity.";
265 ```
266
267 Key principles:
268 - State the agent's role and name clearly (models respond better to named personas)
269 - List specializations explicitly (influences which parts of training are activated)
270 - Set tone expectations: "concise", "practical", "idiomatic"
271 - Avoid verbose instruction lists — they cost tokens on every turn
272
273 ### Streaming tokens to ACP (connecting onde → ACP)
274
275 ```rust
276 // In the prompt handler — cx: &ConnectionTo<Client> is passed in by the builder
277 // (agent-client-protocol 0.13). No mpsc forwarder; send through cx directly.
278 let mut rx = self.engine.stream_message(user_text).await
279 .map_err(|e| Error::new(-32603, e.to_string()))?;
280
281 while let Some(chunk) = rx.recv().await {
282 if !chunk.delta.is_empty() {
283 cx.send_notification(SessionNotification::new(
284 session_id.clone(),
285 SessionUpdate::AgentMessageChunk(
286 ContentChunk::new(ContentBlock::from(chunk.delta)),
287 ),
288 )).ok(); // .ok() — ignore if the client is gone
289 }
290 if chunk.done { break; }
291 }
292
293 Ok(PromptResponse::new(StopReason::EndTurn))
294 ```
295
296 The `PromptResponse` is returned AFTER the stream finishes. The client receives
297 streaming tokens via `session/update` notifications while blocking on the
298 `session/prompt` response. See the `agent-client-protocol` skill for the `cx`
299 (`ConnectionTo<Client>`) model that replaced the old mpsc-channel forwarder.
300
301 > **Note:** siGit's actual `handle_prompt` does *not* stream token-by-token — it
302 > runs a tool-calling loop through an `InferenceBackend` and sends the final text
303 > in one `AgentMessageChunk`. The streaming pattern above still applies if you
304 > want incremental output. See the `tool-calling` skill for the backend loop.
305
306 ---
307
308 ## Extracting text from ACP `PromptRequest`
309
310 ACP prompts can contain text, images, resource links, etc. For a text-only
311 coding agent:
312
313 ```rust
314 let user_text: String = args.prompt.iter()
315 .filter_map(|block| match block {
316 ContentBlock::Text(t) => Some(t.text.as_str()),
317 // Skip images, resource links, embedded resources for now
318 _ => None,
319 })
320 .collect::<Vec<_>>()
321 .join("\n");
322 ```
323
324 For resource context (e.g. open files provided by Zed) — note the variant is
325 `TextResourceContents`, not `Text`:
326 ```rust
327 ContentBlock::Resource(r) => match &r.resource {
328 EmbeddedResourceResource::TextResourceContents(t) => Some(t.text.as_str()),
329 EmbeddedResourceResource::BlobResourceContents(_) => None,
330 _ => None,
331 },
332 ```
333 siGit also handles `ContentBlock::ResourceLink` (a `file://` reference it reads
334 from disk, including `#L<start>:<end>` line-range fragments). See the
335 `tool-calling` skill.
336
337 ---
338
339 ## `ChatEngine` threading model
340
341 - Internally uses `Arc<tokio::sync::Mutex<Option<LoadedModel>>>` — `Send + Sync`.
342 - Safe to wrap in `Arc<ChatEngine>` and share across tasks.
343 - `stream_message()` spawns a `tokio::spawn` background task internally — the
344 mistralrs model must be `Send`, which it is on all supported platforms.
345 - **`block_in_place` trap:** `load_gguf_model` calls `tokio::task::block_in_place`
346 internally, which panics unless it's on a multi-threaded runtime worker. Run
347 model loads on a dedicated `std::thread` with its own `tokio::runtime::Runtime`
348 and signal back via `AtomicBool`/`oneshot`. siGit does exactly this in both ACP
349 and TUI modes — see the `agent-client-protocol` and `tool-calling` skills.
350
351 ---
352
353 ## First-run model download
354
355 On first use, onde downloads the GGUF model from HuggingFace Hub:
356 - Requires internet connectivity
357 - Cached at `~/.cache/huggingface/` (or `HF_HUB_CACHE` env var)
358 - `HF_TOKEN` env var needed for gated models (public Qwen models don't need it)
359 - Subsequent runs load from disk cache — fast
360
361 For sandboxed environments (iOS, tvOS, Android):
362 - Set `HF_HOME` and `HF_HUB_CACHE` to a path inside the app container
363 - Do this BEFORE calling any ChatEngine method
364 - See `onde/docs/swift-package.md` for `setupInferenceEnvironment()` pattern
365
366 ---
367
368 ## Common mistakes
369
370 1. **Calling `load_gguf_model` twice** without checking `is_loaded()` first →
371 `InferenceError::AlreadyLoaded`. Always guard with `is_loaded().await`.
372
373 2. **Blocking on the stream after the channel is closed** → the stream naturally
374 ends when the `done` flag is true. Don't `recv()` after `done`.
375
376 3. **Losing `StreamChunk` deltas** when `delta` is empty (whitespace tokens) →
377 always check `!chunk.delta.is_empty()` before sending to avoid empty
378 notifications that waste bandwidth.
379
380 4. **Sharing one `ChatEngine` across parallel prompts** without coordination →
381 the internal Mutex serializes inference, so concurrent prompts queue up.
382 Design for sequential access per engine instance.
383
384 5. **Using `SamplingConfig::default()` for code generation** → prefer
385 `SamplingConfig::coding()` (deterministic, temp=0) for more reliable code output.
386
387 6. **Forgetting that `generate()` doesn't update history** — use it for
388 one-shot enhancements (prompt expansion, code review) that shouldn't pollute
389 the main conversation. Use `send_message()` / `stream_message()` for the
390 primary turn loop.