AGENTS.md · ondeinference/onde · siGit Code & Deploy

@ondeinference / onde

Code Commits Issues Pulls Cloud Agent CI/CD

development

copilot/assess-inference-engine-complexity copilot/fix-onde-npm-sdk-v1-1-2-release development feature/diffusion-vision main

md 617 lines 30 KB

Rendered Raw

1	# Onde Inference — AGENTS.md
2
3	> AI agent reference for the `onde` repository and the `onde-swift` Swift SDK.
4	> Keep this file accurate as the codebase evolves.
5
6	---
7
8	## What Is Onde?
9
10	Onde is an on-device LLM inference SDK family built around a shared Rust core. It targets Apple silicon (iOS, tvOS, macOS) plus Android, Windows, and Linux, and ships SDKs for Rust, Swift, Kotlin Multiplatform, Flutter, and React Native. It wraps [mistral.rs](https://github.com/EricLBuehler/mistral.rs) behind platform-friendly APIs with automatic model selection, HuggingFace Hub downloads, cache management, and Metal acceleration where available.
11
12	- Website: https://ondeinference.com
13	- Rust crate: https://crates.io/crates/onde
14	- Swift package: https://github.com/ondeinference/onde-swift
15	- Maven Central: https://central.sonatype.com/artifact/com.ondeinference/onde-inference
16	- Flutter package: https://pub.dev/packages/onde_inference
17	- React Native package: https://www.npmjs.com/package/@ondeinference/react-native
18	- In production: [Splitfire AB apps on the Apple AppStore](https://apps.apple.com/se/developer/splitfire-ab/id1831430993)
19
20	---
21
22	## Repository Layout
23
24	```
25	onde/
26	├── src/
27	│ ├── lib.rs # Crate root — uniffi::setup_scaffolding!()
28	│ ├── hf_cache.rs # HuggingFace Hub cache: list, download, delete, diagnose, repair
29	│ └── inference/
30	│ ├── mod.rs # Module exports + top-level re-exports
31	│ ├── engine.rs # ChatEngine — Rust-native API (generics, mpsc, tool calling)
32	│ ├── ffi.rs # OndeChatEngine — UniFFI Object (FFI-safe, Arc-wrapped)
33	│ ├── models.rs # Model ID constants + SupportedModelInfo metadata
34	│ ├── token.rs # HF token resolution: build-time literal vs cache file
35	│ └── types.rs # Shared types: ChatMessage, SamplingConfig, InferenceResult, ToolCallInfo, etc.
36	├── sdk/
37	│ ├── dart/ # Flutter/Dart package + example app + FRB Rust bridge
38	│ ├── gem/ # Ruby native extension (Magnus)
39	│ ├── kotlin/ # Kotlin Multiplatform package (Android + JVM)
40	│ ├── react-native/ # Expo module wrapping the Rust core for iOS/Android
41	│ └── python/ # Python bindings (maturin + uniffi)
42	├── generated/ # UniFFI-generated headers and Swift glue (git-ignored)
43	├── uniffi-bindgen/ # Standalone bindgen binary crate (pinned uniffi =0.31.0)
44	├── .github/
45	│ ├── workflows/
46	│ │ ├── release-sdk-swift.yml # CI: tag push → build XCFramework → GitHub Release → update onde-swift
47	│ │ ├── release-sdk-kotlin.yml # CI: tag push → build Android/JVM artifacts → publish to Maven Central
48	│ │ ├── release-sdk-dart.yml # CI: tag push → publish Flutter package to pub.dev
49	│ │ ├── release-sdk-npm.yml # CI: tag push → publish React Native package to npm
50	│ │ └── release-sdk-rust.yml # CI: tag push → publish Rust crate to crates.io
51	│ └── scripts/
52	│ └── build-swift-xcframework.sh # Local/CI XCFramework assembly script
53	├── .cargo/config.toml # Target-specific rustflags (fp16, linker overrides)
54	├── Cargo.toml # Platform-conditional mistralrs deps
55	├── build.rs # tvOS ___chkstk_darwin assembly stub
56	├── scripts/
57	│ └── tvos_chkstk.s # No-op arm64 stub for missing tvOS symbol
58	├── uniffi.toml # UniFFI binding config
59	└── docs/
60	├── dev.md # Developer guide (build, architecture, platform table)
61	├── swift-package.md # Swift API reference + XCFramework build steps
62	├── distribution.md # Release process for all registries
63	└── ruby-gem.md # Ruby gem API reference
64	```
65
66	```
67	onde-swift/ # Swift Package Manager wrapper repo
68	├── Package.swift # Declares OndeFramework.xcframework binary target
69	└── Sources/
70	└── Onde/
71	└── onde.swift # UniFFI-generated Swift glue (do NOT edit manually)
72	```
73
74	---
75
76	## Architecture
77
78	### Layer Diagram
79
80	```
81	┌──────────────────────────────────────────────────────────┐
82	│ Swift (iOS / tvOS / macOS) Rust app / CLI / server │
83	│ import Onde use onde::inference::* │
84	└──────────┬───────────────────────────────┬───────────────┘
85	│ UniFFI FFI bindings │ Direct Rust API
86	▼ ▼
87	┌──────────────────────────┐ ┌─────────────────────────────┐
88	│ OndeChatEngine │ │ ChatEngine │
89	│ (ffi.rs — uniffi::Object│──│ (engine.rs — Rust-native) │
90	│ FFI-safe, Arc<Self>) │ │ generics, mpsc channels │
91	└──────────┬───────────────┘ └─────────────┬───────────────┘
92	│ │
93	└────────────────┬───────────────┘
94	▼
95	┌────────────────────────┐
96	│ mistralrs::Model │
97	│ GgufModelBuilder │
98	│ Metal / CUDA / CPU │
99	└────────────────────────┘
100	```
101
102	### Key Design Rules
103
104	1. `ChatEngine` (`engine.rs`) owns all Rust-idiomatic logic: `impl Into<String>`, `tokio::sync::mpsc::Receiver`, etc. Never add UniFFI annotations here.
105	2. `OndeChatEngine` (`ffi.rs`) is a thin `Arc`-wrapped UniFFI `Object` with concrete, FFI-safe signatures. It delegates every call to the inner `ChatEngine`.
106	3. `StreamChunkListener` is a `#[uniffi::export(callback_interface)]` trait — it must be passed as a parameter to free functions (not Object methods) because UniFFI 0.31 does not support callback_interface on Object methods.
107	4. Free functions in `ffi.rs` (e.g. `stream_chat_message`, `default_model_config`, `user_message`) are exported with `#[uniffi::export]` and generate top-level Swift functions.
108	5. `uniffi::setup_scaffolding!()` lives in `lib.rs`. Never call `uniffi::generate_scaffolding!()` from `build.rs` for the proc-macro path.
109
110	---
111
112	## Platform Support
113
114	\| Platform \| `target_os` \| GPU Backend \| Default Model \| mistralrs features \|
115	\| -------- \| ----------- \| ----------------- \| ------------- \| ------------------ \|
116	\| macOS \| `macos` \| Metal \| Qwen 2.5 Coder 3B (~1.93 GB) \| `["metal"]` \|
117	\| iOS \| `ios` \| Metal \| Qwen 2.5 Coder 1.5B (~941 MB) \| `["metal"]` \|
118	\| tvOS \| `tvos` \| Metal \| Qwen 2.5 Coder 1.5B (~941 MB) \| `["metal"]` \|
119	\| Android \| `android` \| CPU (candle) \| Qwen 2.5 Coder 1.5B (~941 MB) \| `[]` + hf-hub \|
120	\| Windows \| `windows` \| CPU (CUDA in CI) \| Qwen 2.5 Coder 3B (~1.93 GB) \| `[]` \|
121	\| Linux \| `linux` \| CPU (CUDA in CI) \| Qwen 2.5 Coder 3B (~1.93 GB) \| `[]` \|
122
123	- Current default routing: `platform_default()` now prefers the Coder variants, 1.5B on mobile and 3B on desktop.
124	- iOS / tvOS memory constraint: iOS gives apps about 2–3 GB. The 3B model (~1.93 GB) can still cause OOM on constrained devices, so mobile defaults stay on the 1.5B model.
125	- tvOS tier-3 target: requires `cargo +nightly -Z build-std`. Stable toolchain cannot build tvOS targets.
126	- Android `hf_hub`: `dirs::home_dir()` panics in the Android sandbox. `hf-hub` is added as an explicit dependency on Android so `HF_HOME` can be seeded programmatically via `hf_hub::api::tokio::ApiBuilder`.
127
128	---
129
130	## Supported Models
131
132	All model constants live in `src/inference/models.rs`. When adding a new model:
133
134	1. Add `pub const` entries for the HF repo ID, GGUF filename, and, on Android, `TOK_MODEL_ID` when needed.
135	2. Add the repo ID to `SUPPORTED_MODELS` so `list_local_hf_models` filters it.
136	3. Add a `SupportedModelInfo` entry to `SUPPORTED_MODEL_INFO` with accurate `expected_size_bytes` from the HF API `siblings[].size`.
137	4. Add a constructor to `GgufModelConfig` in `engine.rs`.
138	5. Export a free function in `ffi.rs` for UniFFI consumers when the model should be reachable from Swift and Kotlin.
139	6. If the GGUF does not ship with a built-in chat template, set `chat_template` explicitly. DeepSeek Coder is the current example.
140
141	### Current Models
142
143	\| Model \| Repo \| File \| Size \| Platforms \|
144	\|-------\|------\|------\|------\|-----------\|
145	\| Qwen 2.5 1.5B Instruct (GGUF Q4_K_M) \| `bartowski/Qwen2.5-1.5B-Instruct-GGUF` \| `Qwen2.5-1.5B-Instruct-Q4_K_M.gguf` \| ~941 MB \| All platforms \|
146	\| Qwen 2.5 3B Instruct (GGUF Q4_K_M) \| `bartowski/Qwen2.5-3B-Instruct-GGUF` \| `Qwen2.5-3B-Instruct-Q4_K_M.gguf` \| ~1.93 GB \| All platforms \|
147	\| Qwen 2.5 Coder 1.5B Instruct (GGUF Q4_K_M) \| `bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF` \| `Qwen2.5-Coder-1.5B-Instruct-Q4_K_M.gguf` \| ~941 MB \| All platforms (mobile default) \|
148	\| Qwen 2.5 Coder 3B Instruct (GGUF Q4_K_M) \| `bartowski/Qwen2.5-Coder-3B-Instruct-GGUF` \| `Qwen2.5-Coder-3B-Instruct-Q4_K_M.gguf` \| ~1.93 GB \| All platforms (desktop default) \|
149	\| Qwen 2.5 Coder 7B Instruct (GGUF Q4_K_M) \| `bartowski/Qwen2.5-Coder-7B-Instruct-GGUF` \| `Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf` \| ~4.4 GB \| Higher-memory devices \|
150	\| Qwen 3 0.6B (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-0.6B-GGUF` \| `Qwen_Qwen3-0.6B-Q4_K_M.gguf` \| ~0.5 GB \| All platforms \|
151	\| Qwen 3 1.7B (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-1.7B-GGUF` \| `Qwen_Qwen3-1.7B-Q4_K_M.gguf` \| ~1.3 GB \| All platforms \|
152	\| Qwen 3 4B (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-4B-GGUF` \| `Qwen_Qwen3-4B-Q4_K_M.gguf` \| ~2.7 GB \| All platforms \|
153	\| Qwen 3 8B (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-8B-GGUF` \| `Qwen_Qwen3-8B-Q4_K_M.gguf` \| ~5 GB \| Higher-memory devices \|
154	\| Qwen 3 14B (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-14B-GGUF` \| `Qwen_Qwen3-14B-Q4_K_M.gguf` \| ~8.4 GB \| Higher-memory devices \|
155	\| Qwen 3 32B (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-32B-GGUF` \| `Qwen_Qwen3-32B-Q4_K_M.gguf` \| ~19.8 GB \| High-memory desktop (32+ GB) \|
156	\| Qwen 3 4B Instruct 2507 (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF` \| `Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf` \| ~2.5 GB \| All platforms (latest non-thinking 4B) \|
157	\| Qwen 3 4B Thinking 2507 (GGUF Q4_K_M) \| `bartowski/Qwen_Qwen3-4B-Thinking-2507-GGUF` \| `Qwen_Qwen3-4B-Thinking-2507-Q4_K_M.gguf` \| ~2.5 GB \| All platforms (latest reasoning 4B) \|
158	\| Qwen 3 30B-A3B Instruct 2507 (GGUF Q4_K_M, MoE) \| `bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF` \| `Qwen_Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf` \| ~18.6 GB \| High-memory desktop (32+ GB), `qwen3moe` \|
159	\| DeepSeek Coder 6.7B Instruct (GGUF Q4_K_M) \| `bartowski/deepseek-coder-6.7b-instruct-GGUF` \| `deepseek-coder-6.7b-instruct-Q4_K_M.gguf` \| ~3.8 GB \| Higher-memory devices, custom chat template \|
160	\| Qwen 2.5 Coder 7B Instruct (ISQ) \| `Qwen/Qwen2.5-Coder-7B-Instruct` \| safetensors (ISQ in-situ) \| ~8 GB \| macOS (ISQ pipeline) \|
161
162	---
163
164	## Key Types (Rust ↔ Swift)
165
166	\| Rust Type \| Swift Type \| Notes \|
167	\|-----------\|------------\|-------\|
168	\| `ChatRole` \| `enum ChatRole` \| `.system`, `.user`, `.assistant` \|
169	\| `ChatMessage` \| `struct ChatMessage` \| `role: ChatRole`, `content: String` \|
170	\| `SamplingConfig` \| `struct SamplingConfig` \| All fields `Optional` \|
171	\| `GgufModelConfig` \| `struct GgufModelConfig` \| `modelId`, `files`, `tokModelId?`, `displayName`, `approxMemory`, `chatTemplate?` \|
172	\| `IsqModelConfig` \| `struct IsqModelConfig` \| `modelId`, `isqBits: UInt8`, `displayName`, `approxMemory` \|
173	\| `InferenceResult` \| `struct InferenceResult` \| `text`, `durationSecs`, `durationDisplay`, `finishReason`, `toolCalls` \|
174	\| `ToolCallInfo` \| `struct ToolCallInfo` \| Structured tool call request emitted by the model \|
175	\| `StreamChunk` \| `struct StreamChunk` \| `delta`, `done`, `finishReason?` \|
176	\| `EngineStatus` \| `enum EngineStatus` \| `.unloaded`, `.loading`, `.ready`, `.generating`, `.error` \|
177	\| `EngineInfo` \| `struct EngineInfo` \| `status`, `modelName?`, `approxMemory?`, `historyLength: UInt64` \|
178	\| `InferenceError` \| `enum InferenceError: Error` \| `noModelLoaded`, `alreadyLoaded`, `modelBuild`, `inference`, `cancelled`, `other` \|
179	\| `OndeChatEngine` \| `class OndeChatEngine` \| Thread-safe; `Arc`-backed; constructed with `OndeChatEngine()` \|
180	\| `StreamChunkListener` \| `protocol StreamChunkListener` \| Implement `onChunk(chunk:) -> Bool` \|
181
182	---
183
184	## HuggingFace Token
185
186	`src/inference/token.rs` resolves the HF token in priority order:
187
188	1. Build-time literal (`HF_TOKEN` env var baked in via `option_env!`) — required for iOS/tvOS (no filesystem token possible).
189	2. On-disk cache (`~/.cache/huggingface/token`) — works on macOS after `mistralrs login`.
190
191	Set `HF_TOKEN` before building:
192
193	```bash
194	export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
195	cargo build --release
196	```
197
198	On sandboxed platforms, your app's setup function must also configure `HF_HOME`, `HF_HUB_CACHE`, and `TMPDIR` to point inside the app container before any `OndeChatEngine` method is called. See `docs/swift-package.md` for the full `setupInferenceEnvironment()` Swift snippet.
199
200	---
201
202	## Building (Rust)
203
204	### Prerequisites
205
206	```bash
207	# Stable toolchain (macOS, iOS, Android, Windows, Linux)
208	rustup toolchain install stable
209
210	# Nightly toolchain (tvOS tier-3 targets only)
211	rustup toolchain install nightly
212	rustup component add rust-src --toolchain nightly
213
214	# Apple targets (stable)
215	rustup target add aarch64-apple-ios aarch64-apple-ios-sim
216	rustup target add aarch64-apple-darwin x86_64-apple-darwin
217	```
218
219	### Commands
220
221	```bash
222	# Verify compilation (macOS host)
223	cargo check
224
225	# Run all tests
226	cargo test
227
228	# Run inference module tests only
229	cargo test inference::
230
231	# Run FFI wrapper tests only
232	cargo test inference::ffi::
233
234	# Clippy lint
235	cargo clippy
236
237	# Format
238	cargo fmt
239	```
240
241	### tvOS
242
243	tvOS targets require nightly and `-Z build-std`:
244
245	```bash
246	cargo +nightly rustc -Z build-std \
247	--target aarch64-apple-tvos --release --lib --crate-type staticlib
248	```
249
250	---
251
252	## Building the Swift XCFramework
253
254	The XCFramework bundles staticlibs for iOS device, iOS simulator, tvOS device, tvOS simulator, and macOS into a single distributable that `Package.swift` references.
255
256	### Quick build (local or CI)
257
258	```bash
259	.github/scripts/build-swift-xcframework.sh
260	```
261
262	Output lands in `dist/swift/`:
263	- `OndeFramework.xcframework/` — the framework tree
264	- `OndeFramework.xcframework.zip` — zipped for SPM remote binary
265	- `OndeFramework.checksum.txt` — SHA-256 for `Package.swift`
266	- `version.txt` — semver from `Cargo.toml`
267	- `Package/Sources/Onde/onde.swift` — generated UniFFI Swift glue
268
269	### Manual steps
270
271	```bash
272	# 1. Build the uniffi-bindgen binary (pinned to uniffi =0.31.0)
273	cargo build --manifest-path uniffi-bindgen/Cargo.toml --release
274	BINDGEN=uniffi-bindgen/target/release/uniffi-bindgen
275
276	# 2. Compile staticlibs per target (use +1.92.0 or current stable)
277	cargo +stable rustc --target aarch64-apple-ios --release --lib --crate-type staticlib
278	cargo +stable rustc --target aarch64-apple-ios-sim --release --lib --crate-type staticlib
279	cargo +stable rustc --target aarch64-apple-darwin --release --lib --crate-type staticlib
280	cargo +nightly rustc -Z build-std --target aarch64-apple-tvos --release --lib --crate-type staticlib
281	cargo +nightly rustc -Z build-std --target aarch64-apple-tvos-sim --release --lib --crate-type staticlib
282
283	# 3. Generate Swift bindings from the iOS arm64 slice
284	$BINDGEN generate target/aarch64-apple-ios/release/libonde.a \
285	--language swift \
286	--out-dir sdk/Onde/Sources/Onde \
287	--config uniffi.toml
288
289	# 4. Assemble the XCFramework (see build-swift-xcframework.sh for full xcodebuild invocation)
290	```
291
292	### Updating `onde-swift` after a new release
293
294	This is fully automated by CI. Publishing a GitHub Release on `onde` triggers the workflow, which:
295
296	1. Builds the XCFramework and attaches it to the release.
297	2. Rewrites `onde-swift/Package.swift` with the release `url:` + `checksum:`.
298	3. Copies the freshly generated `onde.swift` into `onde-swift/Sources/Onde/`.
299	4. Commits, tags, and pushes `onde-swift` — Swift Package Index picks it up automatically.
300
301	Manual intervention is only needed if `ONDE_SWIFT_PAT` has expired or the `onde-swift` push fails.
302
303	---
304
305	## Swift SDK (`onde-swift`)
306
307	### Package.swift structure
308
309	```
310	onde-swift/
311	├── Package.swift # Declares OndeFramework binary target + Onde wrapper target
312	└── Sources/Onde/
313	└── onde.swift # UniFFI-generated glue (NEVER edit manually)
314	```
315
316	`Package.swift` has two targets:
317
318	- `OndeFramework` — `.binaryTarget` using `url:` + `checksum:` for distribution. For local development, swap to the `path:` form documented in the `Package.swift` header comment and run `build-swift-xcframework.sh` first.
319	- `Onde` — `.target` depending on `OndeFramework`, used as the public import name in Swift.
320
321	### Swift API Quick Reference
322
323	```swift
324	import Onde
325
326	// Create engine
327	let engine = OndeChatEngine()
328
329	// Load the platform default model
330	let elapsed = try await engine.loadDefaultModel(
331	systemPrompt: "You are a helpful assistant.",
332	sampling: nil
333	)
334
335	// Or load the model assigned to your app in the Onde dashboard
336	let assignedElapsed = try await engine.loadAssignedModel(
337	appId: "your-app-id",
338	appSecret: "your-app-secret",
339	systemPrompt: "You are a helpful assistant.",
340	sampling: nil
341	)
342
343	// Multi-turn chat
344	let result = try await engine.sendMessage(message: "Hello!")
345	print(result.text)
346	print(result.toolCalls)
347
348	// Streaming (free function — callback_interface constraint in UniFFI 0.31)
349	class Handler: StreamChunkListener {
350	func onChunk(chunk: StreamChunk) -> Bool {
351	print(chunk.delta, terminator: "")
352	return !chunk.done
353	}
354	}
355	try await streamChatMessage(engine: engine, message: "Tell me a story.", listener: Handler())
356
357	// One-shot (does NOT modify conversation history)
358	let enhanced = try await engine.generate(
359	messages: [userMessage(content: "Expand: a cat in space")],
360	sampling: deterministicSamplingConfig()
361	)
362
363	// Status
364	let info = await engine.info() // EngineInfo
365
366	// History management
367	let history = await engine.history()
368	let removed = await engine.clearHistory()
369	await engine.pushHistory(message: userMessage(content: "..."))
370
371	// Cleanup
372	await engine.unloadModel()
373	```
374
375	### Free Functions
376
377	\| Function \| Returns \| Notes \|
378	\|----------\|---------\|-------\|
379	\| `defaultModelConfig()` \| `GgufModelConfig` \| Platform-aware Coder default (1.5B on iOS/tvOS/Android, 3B on desktop) \|
380	\| `qwen251_5bConfig()` \| `GgufModelConfig` \| Forces Qwen 2.5 1.5B regardless of platform \|
381	\| `qwen253bConfig()` \| `GgufModelConfig` \| Forces Qwen 2.5 3B regardless of platform \|
382	\| `qwen306bConfig()` \| `GgufModelConfig` \| Qwen 3 0.6B (~0.5 GB) \|
383	\| `qwen317bConfig()` \| `GgufModelConfig` \| Qwen 3 1.7B (~1.3 GB) \|
384	\| `qwen34bConfig()` \| `GgufModelConfig` \| Qwen 3 4B (~2.7 GB) \|
385	\| `qwen38bConfig()` \| `GgufModelConfig` \| Qwen 3 8B (~5 GB) \|
386	\| `qwen314bConfig()` \| `GgufModelConfig` \| Qwen 3 14B (~8.4 GB) \|
387	\| `qwen332bConfig()` \| `GgufModelConfig` \| Qwen 3 32B (~19.8 GB) \|
388	\| `qwen34bInstruct2507Config()` \| `GgufModelConfig` \| Qwen 3 4B Instruct 2507 — latest non-thinking 4B (~2.5 GB) \|
389	\| `qwen34bThinking2507Config()` \| `GgufModelConfig` \| Qwen 3 4B Thinking 2507 — latest reasoning 4B (~2.5 GB) \|
390	\| `qwen330bA3bInstruct2507Config()` \| `GgufModelConfig` \| Qwen 3 30B-A3B Instruct 2507 MoE (~18.6 GB) \|
391	\| `defaultSamplingConfig()` \| `SamplingConfig` \| temp=0.7, top_p=0.95, max_tokens=512 \|
392	\| `deterministicSamplingConfig()` \| `SamplingConfig` \| temp=0.0, greedy \|
393	\| `mobileSamplingConfig()` \| `SamplingConfig` \| temp=0.7, max_tokens=128 \|
394	\| `systemMessage(content:)` \| `ChatMessage` \| `.system` role \|
395	\| `userMessage(content:)` \| `ChatMessage` \| `.user` role \|
396	\| `assistantMessage(content:)` \| `ChatMessage` \| `.assistant` role \|
397	\| `streamChatMessage(engine:message:listener:)` \| `async throws` \| Streaming via callback \|
398
399	---
400
401	## UniFFI Conventions
402
403	- Version pin: `uniffi = "=0.31.0"` everywhere — the `onde` crate, `uniffi-bindgen/`, and `[build-dependencies]` must all use the same version. Mixing versions causes bindgen panics.
404	- `uniffi::setup_scaffolding!()` in `lib.rs` — proc-macro approach, no UDL file needed.
405	- Object methods use `#[uniffi::export]` on the `impl OndeChatEngine` block.
406	- Callback interfaces (`StreamChunkListener`) must be parameters of free functions only, not Object methods.
407	- `Arc<Self>` is the return type for `#[uniffi::constructor]`. UniFFI automatically handles this.
408	- Async: all async exported methods use `tokio` runtime (enabled via `uniffi = { features = ["tokio"] }`).
409	- `uniffi.toml`: lives at the crate root. Adjust renaming or namespace settings there before regenerating.
410
411	---
412
413	## HuggingFace Cache (`hf_cache.rs`)
414
415	The `hf_cache` module manages the on-device model cache, exposed via UniFFI to Swift / Kotlin as-needed.
416
417	Key public functions:
418
419	\| Function \| Description \|
420	\|----------\|-------------\|
421	\| `list_local_hf_models()` \| List downloaded models that Onde supports \|
422	\| `list_supported_hf_models()` \| All supported models with download status \|
423	\| `download_model(model_id, progress_callback)` \| Download a model with progress reporting \|
424	\| `delete_local_hf_model(model_id)` \| Remove a model from the local cache \|
425	\| `diagnose_hf_cache()` \| Inspect the cache for corruption \|
426	\| `repair_hf_cache_symlinks()` \| Fix broken symlinks in the HF cache layout \|
427	\| `model_cache_path(model_id)` \| Resolve the filesystem path for a model \|
428	\| `clean_stale_lock_files()` \| Remove leftover `.lock` files from interrupted downloads \|
429
430	`ModelDownloadProgress` carries `downloaded_bytes`, `total_bytes`, `progress` (0.0–1.0), and `done`.
431
432	---
433
434	## Sampling Presets
435
436	\| Preset \| `temperature` \| `top_p` \| `max_tokens` \| Use Case \|
437	\|--------\|--------------\|---------\|--------------\|----------\|
438	\| `SamplingConfig::default()` \| 0.7 \| 0.95 \| 512 \| General creative chat \|
439	\| `SamplingConfig::deterministic()` \| 0.0 \| — \| 512 \| Reproducible / coding \|
440	\| `SamplingConfig::mobile()` \| 0.7 \| 0.95 \| 128 \| Memory/latency constrained \|
441	\| `SamplingConfig::coding()` \| 0.0 \| — \| 512 \| Code generation \|
442	\| `SamplingConfig::coding_mobile()` \| 0.0 \| — \| 128 \| Code on mobile \|
443
444	---
445
446	## Testing
447
448	```bash
449	# All tests (requires a macOS host with Metal)
450	cargo test
451
452	# Unit tests only (no model downloads)
453	cargo test inference::
454	cargo test inference::ffi::
455	cargo test inference::types::
456
457	# hf_cache module
458	cargo test hf_cache::
459
460	# Clippy (treat warnings as errors)
461	cargo clippy -- -D warnings
462	```
463
464	Tests that require model downloads are integration tests and not run by default in CI. Unit tests verify:
465	- Type constructors and `Display` implementations
466	- `SamplingConfig` presets
467	- `GgufModelConfig` constructors and `platform_default()` routing
468	- `OndeChatEngine` lifecycle: `new()` starts unloaded, `send_message` without model returns `InferenceError::NoModelLoaded`, `clear_history` on empty returns 0, `unload_model` on empty returns `nil`.
469
470	---
471
472	## Known Issues
473
474	### `___chkstk_darwin` linker error on tvOS
475
476	`aws-lc-sys` (transitive via `reqwest → rustls → aws-lc-rs`) references `___chkstk_darwin`, a stack probing symbol that tvOS does not export. `build.rs` compiles `scripts/tvos_chkstk.s` (a no-op `ret` stub) via the `cc` crate to satisfy the linker. Do not delete `scripts/tvos_chkstk.s`.
477
478	Affects: tvOS only. macOS, iOS, Android, Windows, Linux are unaffected.
479
480	### Metal Toolchain missing (Xcode 26+)
481
482	On Xcode 26+, the Metal compiler is a separate download. If you see empty `.metallib` files (92 bytes) or `"Error while loading function: fused_glu_float"` at runtime:
483
484	```bash
485	xcodebuild -downloadComponent MetalToolchain
486	cargo clean -p mistralrs-quant
487	cargo check
488	```
489
490	### Android `home_dir` sandbox panic
491
492	`dirs::home_dir()` panics in the Android sandbox. The crate-level `Cargo.toml` adds `hf-hub` as an explicit Android dependency so `HF_HOME` can be seeded via `ApiBuilder` before any hub request. Never call `home::home_dir()` or `dirs::home_dir()` on Android.
493
494	---
495
496	## Code Conventions
497
498	### Rust
499
500	- Error handling: `anyhow` for application-level errors inside `engine.rs`; `thiserror` for the `InferenceError` enum (exported via UniFFI). Never `.unwrap()` or `.expect()` in non-test code.
501	- Async: `tokio` runtime. All async functions in `OndeChatEngine` are `pub async fn`.
502	- Logging: `log` crate macros (`log::debug!`, `log::info!`, `log::warn!`, `log::error!`). No `println!` in library code.
503	- Platform gating: use `#[cfg(target_os = "...")]` blocks. Match `Cargo.toml`'s target-conditional dependency sections.
504	- Re-exports: `mistralrs`, `hf_hub`, and `mistralrs_core` are re-exported from `lib.rs` for downstream Rust consumers. Keep these re-exports in sync with what's actually available per platform.
505	- No `mod.rs`: use named files (`inference/engine.rs`) not `inference/mod.rs` — except that `inference/mod.rs` exists and is the intentional module root for the `inference` module.
506
507	### Swift / SDK
508
509	- Never manually edit `onde-swift/Sources/Onde/onde.swift` — it is generated by `uniffi-bindgen`. Regenerate by running the build script.
510	- `Package.swift` binary target uses `url:` + `checksum:` in the committed form. For local development, swap to `path:` (instructions are in the file header). Never commit the `path:` form — CI overwrites it on every release via `release-sdk-swift.yml`.
511	- iOS/tvOS sandbox: always call `setupInferenceEnvironment()` at app launch before any `OndeChatEngine` call.
512
513	---
514
515	## CI / Release Workflow
516
517	### `release-sdk-swift.yml`
518
519	Triggered on semver tag push or `workflow_dispatch`. Runs on `macos-15`.
520
521	Steps:
522	1. Install stable Rust with iOS/macOS targets.
523	2. Install nightly Rust with `rust-src` for tvOS `-Z build-std`.
524	3. Run `build-swift-xcframework.sh`.
525	4. Read `version.txt` and `OndeFramework.checksum.txt` into step outputs.
526	5. Validate that the tag matches `Cargo.toml`.
527	6. Upload CI artifacts: zip, checksum, version, generated `onde.swift`.
528	7. On tag push, create a GitHub Release with the zip and checksum attached.
529	8. On tag push, check out `ondeinference/onde-swift` using `ONDE_SWIFT_PAT`.
530	9. On tag push, rewrite the `.binaryTarget` in `onde-swift/Package.swift` with the release URL and checksum.
531	10. On tag push, copy the generated `onde.swift` into `onde-swift/Sources/Onde/`.
532	11. On tag push, commit both files, tag the commit with the version, and push to `ondeinference/onde-swift`.
533
534	#### Required secret
535
536	`ONDE_SWIFT_PAT` — a GitHub Personal Access Token with `contents: write` scope on the `ondeinference/onde-swift` repository.
537
538	### `release-sdk-kotlin.yml`
539
540	Triggered on semver tag push or `workflow_dispatch`. Runs on `macos-15` because it builds both Android artifacts and the macOS Apple Silicon JVM native library.
541
542	Steps:
543	1. Install stable Rust and the `aarch64-apple-darwin` target.
544	2. Install Android SDK / NDK and auto-discover the latest installed 29.x NDK for `cargo-ndk`.
545	3. Build Android JNI libraries for all supported ABIs.
546	4. Build the macOS Apple Silicon `libonde.dylib` used by the JVM target.
547	5. Run Gradle publication tasks for the Kotlin Multiplatform package.
548	6. Validate that the tag matches `sdk/kotlin/gradle.properties` `VERSION_NAME`.
549	7. On tag push, publish `com.ondeinference:onde-inference` to Maven Central.
550
551	#### Required secrets
552
553	- `ORG_GRADLE_PROJECT_MAVENCENTRALUSERNAME`
554	- `ORG_GRADLE_PROJECT_MAVENCENTRALPASSWORD`
555	- `ORG_GRADLE_PROJECT_SIGNINGKEYID`
556	- `ORG_GRADLE_PROJECT_SIGNINGKEY`
557	- `ORG_GRADLE_PROJECT_SIGNINGPASSWORD`
558
559	### `release-sdk-dart.yml`
560
561	Triggered on semver tag push or `workflow_dispatch`.
562
563	- Publishes `onde_inference` to pub.dev.
564	- Version must match the Dart package version.
565	- The example app is part of release validation.
566
567	### `release-sdk-npm.yml`
568
569	Triggered on semver tag push or `workflow_dispatch`.
570
571	- Publishes `@ondeinference/react-native` to npm.
572	- Builds the Expo module package and validates the published version.
573
574	### `release-sdk-rust.yml`
575
576	Triggered on semver tag push or `workflow_dispatch`.
577
578	- Publishes `onde` to crates.io.
579	- Uses registry dependencies, not git refs, because `cargo publish` strips git patches.
580
581	### Release Process (end-to-end)
582
583	1. Bump versions in every affected package manifest:
584	- `Cargo.toml`
585	- `sdk/kotlin/gradle.properties`
586	- `sdk/dart/pubspec.yaml`
587	- `sdk/react-native/package.json`
588	2. Merge release branches with `--no-ff`. Do not fast-forward multi-commit feature branches.
589	3. Commit, tag, and push the release.
590	4. CI publishes all five SDKs:
591	- crates.io (`onde`)
592	- Swift Package Index / GitHub Releases (`onde-swift`)
593	- Maven Central (`com.ondeinference:onde-inference`)
594	- pub.dev (`onde_inference`)
595	- npm (`@ondeinference/react-native`)
596	5. Verify the registries after publish, especially Maven Central indexing and the generated Swift release assets.
597
598	> Note: `onde-swift/Package.swift` is updated automatically by CI. For local development only, switch it to the `path:` form documented in the file header, then switch it back before committing.
599
600	---
601
602	## Distribution Registry Summary
603
604	\| Registry \| Name \| Import \|
605	\|----------\|------\|--------\|
606	\| crates.io \| `onde` \| `onde = "1.x"` \|
607	\| Swift Package Index \| `onde-swift` (org: `ondeinference`) \| `import Onde` \|
608	\| Maven Central \| `com.ondeinference:onde-inference` \| Gradle `implementation("com.ondeinference:onde-inference:<version>")` \|
609	\| pub.dev \| `onde_inference` \| `import 'package:onde_inference/onde_inference.dart';` \|
610	\| npm \| `@ondeinference/react-native` \| `import { OndeChatEngine } from "@ondeinference/react-native"` \|
611	\| PyPI \| `onde-inference` \| `import onde_inference` \|
612	\| RubyGems \| `onde-inference` \| `require 'onde'` \|
613
614	---
615
616	This file is for AI agent and human developer reference. Update it when the architecture, API surface, supported models, or toolchain requirements change.
617	```