@setoelkahfi / sigit
Code Commits Pulls 2 CI/CD
feature/sigit-code-cloud
claude/code-feature-parity-q003hm claude/elegant-carson-l1menh claude/sigit-acp-local-chat-cx6380 claude/sigit-cloud-agent-expansion-reox0a claude/tool-permission-system claude/zen-feynman-0u78dk development feature/agent-tools-multiedit-glob-todos-remember feature/background-commands feature/commit-coauthor-attribution feature/headless-mode feature/init-command feature/load-local-model-explicitly feature/session-persistence-compaction feature/sigit-code-cloud feature/subagent-tool feature/tool-permission-system feature/tui-repo-tabs feature/tui-tabs main release/v1.3.1
md 366 lines 11.7 KB
Rendered Raw
1 ---
2 name: ai-assisted-coding
3 description: Build or maintain AI-assisted coding features in Rust using Onde Inference. Use when working on ChatEngine integration, model loading, streaming inference, history management, sampling config, or local coding-agent architecture.
4 ---
5
6 # Skill: AI-Assisted Coding Agents — Onde Inference Integration
7
8 ## Overview
9
10 Building a local AI coding agent in Rust using Onde Inference as the LLM backend.
11 Onde wraps mistral.rs with a clean API for model loading, history management, and
12 streaming inference across macOS (Metal), iOS, Android, Linux, and Windows.
13
14 Crate: `onde = { path = "../onde" }` or from crates.io when published
15 Repo: https://github.com/ondeinference/onde
16 Docs: https://ondeinference.com
17
18 ---
19
20 ## Onde `ChatEngine` API
21
22 ### Construction and lifecycle
23
24 ```rust
25 use onde::inference::{ChatEngine, GgufModelConfig, SamplingConfig};
26
27 let engine = ChatEngine::new(); // starts unloaded
28 engine.is_loaded().await // -> bool
29 engine.unload_model().await // -> ()
30 ```
31
32 ### Loading a model
33
34 ```rust
35 // Platform-aware default (Qwen 2.5 3B on macOS, 1.5B on iOS/tvOS/Android)
36 let config = GgufModelConfig::platform_default();
37
38 // Load — blocks until model is in memory and on GPU
39 engine
40 .load_gguf_model(
41 config,
42 Some("You are a helpful assistant.".to_string()), // system prompt
43 None, // sampling config (uses SamplingConfig::default() internally)
44 )
45 .await?;
46
47 // AlreadyLoaded error if called twice — check first:
48 if !engine.is_loaded().await {
49 engine.load_gguf_model(...).await?;
50 }
51 ```
52
53 **Model sizes (macOS/Windows/Linux default — Qwen 2.5 3B Q4_K_M):** ~1.93 GB
54 **Model sizes (iOS/tvOS/Android default — Qwen 2.5 1.5B Q4_K_M):** ~941 MB
55 First run downloads from HuggingFace Hub into `~/.cache/huggingface/`.
56
57 ### Blocking (non-streaming) inference
58
59 ```rust
60 let result = engine.send_message("What is Rust's ownership model?").await?;
61 // result: InferenceResult
62 println!("{}", result.text);
63 println!("took {}", result.duration_display); // e.g. "3.2s"
64 ```
65
66 `send_message` appends both the user message and assistant reply to conversation
67 history automatically.
68
69 ### Streaming inference
70
71 ```rust
72 let mut rx: tokio::sync::mpsc::Receiver<StreamChunk> =
73 engine.stream_message("Tell me a story.").await?;
74
75 while let Some(chunk) = rx.recv().await {
76 if !chunk.delta.is_empty() {
77 print!("{}", chunk.delta); // partial token text
78 }
79 if chunk.done {
80 // chunk.finish_reason: Option<String> — e.g. "stop", "length"
81 break;
82 }
83 }
84 ```
85
86 `StreamChunk` fields:
87 - `delta: String` — the new token(s) in this chunk
88 - `done: bool` — true on the last chunk
89 - `finish_reason: Option<String>` — present on final chunk only
90
91 History is updated automatically after the stream completes.
92
93 ### One-shot generation (no history side-effects)
94
95 ```rust
96 use onde::inference::ChatMessage;
97
98 let result = engine.generate(
99 vec![ChatMessage::user("Expand: a cat in space")],
100 Some(SamplingConfig::deterministic()),
101 ).await?;
102 println!("{}", result.text);
103 // Does NOT modify conversation history
104 ```
105
106 ### History management
107
108 ```rust
109 let history: Vec<ChatMessage> = engine.history().await;
110 let removed: usize = engine.clear_history().await; // returns count cleared
111 engine.push_history(ChatMessage::user("context")).await;
112 engine.set_system_prompt("new system prompt").await;
113 engine.clear_system_prompt().await;
114 ```
115
116 ### Engine status
117
118 ```rust
119 let info: EngineInfo = engine.info().await;
120 // info.status: EngineStatus (Unloaded | Loading | Ready | Generating | Error)
121 // info.model_name: Option<String>
122 // info.approx_memory: Option<String> e.g. "~1.93 GB"
123 // info.history_length: u64
124 ```
125
126 ---
127
128 ## `InferenceError` variants
129
130 ```rust
131 match err {
132 InferenceError::NoModelLoaded => { /* load model first */ }
133 InferenceError::AlreadyLoaded { model_name } => { /* already loaded */ }
134 InferenceError::ModelBuild { reason } => { /* load failure */ }
135 InferenceError::Inference { reason } => { /* runtime inference error */ }
136 InferenceError::Cancelled => { /* was cancelled */ }
137 InferenceError::Other { reason } => { /* unexpected */ }
138 }
139 ```
140
141 Map to ACP errors:
142 ```rust
143 .map_err(|e| agent_client_protocol::Error::new(-32603, e.to_string()))?
144 ```
145
146 ---
147
148 ## `SamplingConfig` presets
149
150 | Preset | temp | top_p | max_tokens | Use case |
151 |--------|------|-------|------------|----------|
152 | `SamplingConfig::default()` | 0.7 | 0.95 | 512 | General chat |
153 | `SamplingConfig::deterministic()` | 0.0 | — | 512 | Code / reproducible |
154 | `SamplingConfig::mobile()` | 0.7 | 0.95 | 128 | Memory-constrained |
155 | `SamplingConfig::coding()` | 0.0 | — | 512 | Code generation |
156 | `SamplingConfig::coding_mobile()` | 0.0 | — | 128 | Code on mobile |
157
158 ---
159
160 ## `GgufModelConfig` constructors
161
162 ```rust
163 GgufModelConfig::platform_default() // auto-selects based on target_os
164 GgufModelConfig::qwen25_1_5b() // force 1.5B
165 GgufModelConfig::qwen25_3b() // force 3B
166 GgufModelConfig::qwen25_coder_1_5b() // coder variant 1.5B
167 GgufModelConfig::qwen25_coder_3b() // coder variant 3B
168 ```
169
170 ---
171
172 ## Adding onde as a Rust library dependency
173
174 ```toml
175 # In your crate's Cargo.toml — onde is a path dep since it's not on crates.io yet
176 onde = { path = "../onde" }
177 ```
178
179 **Important:** `onde` declares `crate-type = ["lib", "cdylib", "staticlib"]`.
180 When used as a Rust library dep, only the `lib` target is compiled. The
181 `cdylib`/`staticlib` targets (used for Swift/Kotlin FFI) are not built. The
182 `uniffi::setup_scaffolding!()` macro generates `#[no_mangle] extern "C"` symbols
183 but these are harmless in a binary context.
184
185 **The `[patch.crates-io]` in onde's Cargo.toml does NOT propagate** to dependents
186 unless they are in the same workspace. The `sysctl` patch is only needed for
187 watchOS; macOS/iOS/Linux work without it.
188
189 **GPU feature selection is automatic** via `target_os` cfg flags in onde's
190 Cargo.toml — you get Metal on macOS/iOS without any extra features in your crate.
191
192 ---
193
194 ## Patterns for coding agents
195
196 ### Single-engine, multi-session via history reset
197
198 For a simple MVP where one session is active at a time:
199
200 ```rust
201 struct MyAgent {
202 engine: Arc<ChatEngine>,
203 active_session: Arc<Mutex<Option<SessionId>>>,
204 }
205
206 // new_session handler:
207 if self.engine.is_loaded().await {
208 self.engine.clear_history().await; // reuse model, fresh conversation
209 } else {
210 self.engine
211 .load_gguf_model(GgufModelConfig::platform_default(), Some(SYSTEM_PROMPT.into()), None)
212 .await?;
213 }
214 ```
215
216 **Why:** Loading the model is expensive (seconds + GB of RAM). Reloading for each
217 session would make the agent feel broken. `clear_history()` resets context in
218 microseconds.
219
220 ### Per-session engines (multiple concurrent sessions)
221
222 When you need truly isolated parallel sessions:
223
224 ```rust
225 use std::collections::HashMap;
226
227 struct MultiSessionAgent {
228 sessions: Arc<Mutex<HashMap<String, Arc<ChatEngine>>>>,
229 }
230
231 // new_session: create and load a new engine per session
232 // prompt: look up session engine, call send_message or stream_message
233 // CAVEAT: each engine holds a separate model copy in GPU memory — expensive!
234 ```
235
236 Better approach for shared GPU memory: use `engine.generate()` (no history
237 side-effects) with an explicitly managed message vec per session.
238
239 ### System prompt design for coding agents
240
241 ```rust
242 const SYSTEM_PROMPT: &str = "\
243 You are <AgentName>, an expert AI coding agent integrated into your editor \
244 via the Agent Client Protocol. You specialize in:
245
246 - Code analysis, writing, and refactoring
247 - Bug hunting and debugging
248 - Git workflows and commit messages
249 - Software architecture and design patterns
250 - Code review and best practices
251
252 Be concise, precise, and practical. Write clean, idiomatic code with brief \
253 explanations. Identify root causes when debugging. Prefer correctness over brevity.";
254 ```
255
256 Key principles:
257 - State the agent's role and name clearly (models respond better to named personas)
258 - List specializations explicitly (influences which parts of training are activated)
259 - Set tone expectations: "concise", "practical", "idiomatic"
260 - Avoid verbose instruction lists — they cost tokens on every turn
261
262 ### Streaming tokens to ACP (connecting onde → ACP)
263
264 ```rust
265 // In Agent::prompt():
266 let mut rx = self.engine.stream_message(user_text).await
267 .map_err(|e| Error::new(-32603, e.to_string()))?;
268
269 while let Some(chunk) = rx.recv().await {
270 if !chunk.delta.is_empty() {
271 self.notification_tx.send(
272 SessionNotification::new(
273 session_id.clone(),
274 SessionUpdate::AgentMessageChunk(
275 ContentChunk::new(ContentBlock::from(chunk.delta)),
276 ),
277 )
278 ).await.ok(); // .ok() — ignore if forwarder is gone
279 }
280 if chunk.done { break; }
281 }
282
283 Ok(PromptResponse::new(StopReason::EndTurn))
284 ```
285
286 The `PromptResponse` is returned AFTER the stream finishes. The client receives
287 streaming tokens via `session/update` notifications while blocking on the
288 `session/prompt` response.
289
290 ---
291
292 ## Extracting text from ACP `PromptRequest`
293
294 ACP prompts can contain text, images, resource links, etc. For a text-only
295 coding agent:
296
297 ```rust
298 let user_text: String = args.prompt.iter()
299 .filter_map(|block| match block {
300 ContentBlock::Text(t) => Some(t.text.as_str()),
301 // Skip images, resource links, embedded resources for now
302 _ => None,
303 })
304 .collect::<Vec<_>>()
305 .join("\n");
306 ```
307
308 For future resource context (e.g. open files provided by Zed):
309 ```rust
310 ContentBlock::Resource(r) => match &r.resource {
311 EmbeddedResourceResource::Text(t) => Some(t.text.as_str()),
312 _ => None,
313 },
314 ```
315
316 ---
317
318 ## `ChatEngine` threading model
319
320 - Internally uses `Arc<tokio::sync::Mutex<Option<LoadedModel>>>` — `Send + Sync`.
321 - Safe to wrap in `Arc<ChatEngine>` and share across tasks.
322 - `stream_message()` spawns a `tokio::spawn` background task internally — the
323 mistralrs model must be `Send`, which it is on all supported platforms.
324 - Calling `stream_message()` from a `!Send` future (e.g. inside a `LocalSet`) is
325 fine — the future itself doesn't hold a `!Send` value across `.await`.
326
327 ---
328
329 ## First-run model download
330
331 On first use, onde downloads the GGUF model from HuggingFace Hub:
332 - Requires internet connectivity
333 - Cached at `~/.cache/huggingface/` (or `HF_HUB_CACHE` env var)
334 - `HF_TOKEN` env var needed for gated models (public Qwen models don't need it)
335 - Subsequent runs load from disk cache — fast
336
337 For sandboxed environments (iOS, tvOS, Android):
338 - Set `HF_HOME` and `HF_HUB_CACHE` to a path inside the app container
339 - Do this BEFORE calling any ChatEngine method
340 - See `onde/docs/swift-package.md` for `setupInferenceEnvironment()` pattern
341
342 ---
343
344 ## Common mistakes
345
346 1. **Calling `load_gguf_model` twice** without checking `is_loaded()` first →
347 `InferenceError::AlreadyLoaded`. Always guard with `is_loaded().await`.
348
349 2. **Blocking on the stream after the channel is closed** → the stream naturally
350 ends when the `done` flag is true. Don't `recv()` after `done`.
351
352 3. **Losing `StreamChunk` deltas** when `delta` is empty (whitespace tokens) →
353 always check `!chunk.delta.is_empty()` before sending to avoid empty
354 notifications that waste bandwidth.
355
356 4. **Sharing one `ChatEngine` across parallel prompts** without coordination →
357 the internal Mutex serializes inference, so concurrent prompts queue up.
358 Design for sequential access per engine instance.
359
360 5. **Using `SamplingConfig::default()` for code generation** → prefer
361 `SamplingConfig::coding()` (deterministic, temp=0) for more reliable code output.
362
363 6. **Forgetting that `generate()` doesn't update history** — use it for
364 one-shot enhancements (prompt expansion, code review) that shouldn't pollute
365 the main conversation. Use `send_message()` / `stream_message()` for the
366 primary turn loop.