|
1
|
# Onde Inference — AGENTS.md |
|
2
|
|
|
3
|
> **AI agent reference for the `onde` repository and the `onde-swift` Swift SDK.** |
|
4
|
> Keep this file accurate as the codebase evolves. |
|
5
|
|
|
6
|
--- |
|
7
|
|
|
8
|
## What Is Onde? |
|
9
|
|
|
10
|
Onde is an **on-device LLM inference SDK family** built around a shared Rust core. It targets Apple silicon (iOS, tvOS, macOS) plus Android, Windows, and Linux, and ships SDKs for Rust, Swift, Kotlin Multiplatform, Flutter, and React Native. It wraps [mistral.rs](https://github.com/EricLBuehler/mistral.rs) behind platform-friendly APIs with automatic model selection, HuggingFace Hub downloads, cache management, and Metal acceleration where available. |
|
11
|
|
|
12
|
- **Website:** https://ondeinference.com |
|
13
|
- **Rust crate:** https://crates.io/crates/onde |
|
14
|
- **Swift package:** https://github.com/ondeinference/onde-swift |
|
15
|
- **Maven Central:** https://central.sonatype.com/artifact/com.ondeinference/onde-inference |
|
16
|
- **Flutter package:** https://pub.dev/packages/onde_inference |
|
17
|
- **React Native package:** https://www.npmjs.com/package/@ondeinference/react-native |
|
18
|
- **In production:** [Splitfire AB apps on the Apple AppStore](https://apps.apple.com/se/developer/splitfire-ab/id1831430993) |
|
19
|
|
|
20
|
--- |
|
21
|
|
|
22
|
## Repository Layout |
|
23
|
|
|
24
|
``` |
|
25
|
onde/ |
|
26
|
├── src/ |
|
27
|
│ ├── lib.rs # Crate root — uniffi::setup_scaffolding!() |
|
28
|
│ ├── hf_cache.rs # HuggingFace Hub cache: list, download, delete, diagnose, repair |
|
29
|
│ └── inference/ |
|
30
|
│ ├── mod.rs # Module exports + top-level re-exports |
|
31
|
│ ├── engine.rs # ChatEngine — Rust-native API (generics, mpsc, tool calling) |
|
32
|
│ ├── ffi.rs # OndeChatEngine — UniFFI Object (FFI-safe, Arc-wrapped) |
|
33
|
│ ├── models.rs # Model ID constants + SupportedModelInfo metadata |
|
34
|
│ ├── token.rs # HF token resolution: build-time literal vs cache file |
|
35
|
│ └── types.rs # Shared types: ChatMessage, SamplingConfig, InferenceResult, ToolCallInfo, etc. |
|
36
|
├── sdk/ |
|
37
|
│ ├── dart/ # Flutter/Dart package + example app + FRB Rust bridge |
|
38
|
│ ├── gem/ # Ruby native extension (Magnus) |
|
39
|
│ ├── kotlin/ # Kotlin Multiplatform package (Android + JVM) |
|
40
|
│ ├── react-native/ # Expo module wrapping the Rust core for iOS/Android |
|
41
|
│ └── python/ # Python bindings (maturin + uniffi) |
|
42
|
├── generated/ # UniFFI-generated headers and Swift glue (git-ignored) |
|
43
|
├── uniffi-bindgen/ # Standalone bindgen binary crate (pinned uniffi =0.31.0) |
|
44
|
├── .github/ |
|
45
|
│ ├── workflows/ |
|
46
|
│ │ ├── release-sdk-swift.yml # CI: tag push → build XCFramework → GitHub Release → update onde-swift |
|
47
|
│ │ ├── release-sdk-kotlin.yml # CI: tag push → build Android/JVM artifacts → publish to Maven Central |
|
48
|
│ │ ├── release-sdk-dart.yml # CI: tag push → publish Flutter package to pub.dev |
|
49
|
│ │ ├── release-sdk-npm.yml # CI: tag push → publish React Native package to npm |
|
50
|
│ │ └── release-sdk-rust.yml # CI: tag push → publish Rust crate to crates.io |
|
51
|
│ └── scripts/ |
|
52
|
│ └── build-swift-xcframework.sh # Local/CI XCFramework assembly script |
|
53
|
├── .cargo/config.toml # Target-specific rustflags (fp16, linker overrides) |
|
54
|
├── Cargo.toml # Platform-conditional mistralrs deps |
|
55
|
├── build.rs # tvOS ___chkstk_darwin assembly stub |
|
56
|
├── scripts/ |
|
57
|
│ └── tvos_chkstk.s # No-op arm64 stub for missing tvOS symbol |
|
58
|
├── uniffi.toml # UniFFI binding config |
|
59
|
└── docs/ |
|
60
|
├── dev.md # Developer guide (build, architecture, platform table) |
|
61
|
├── swift-package.md # Swift API reference + XCFramework build steps |
|
62
|
├── distribution.md # Release process for all registries |
|
63
|
└── ruby-gem.md # Ruby gem API reference |
|
64
|
``` |
|
65
|
|
|
66
|
``` |
|
67
|
onde-swift/ # Swift Package Manager wrapper repo |
|
68
|
├── Package.swift # Declares OndeFramework.xcframework binary target |
|
69
|
└── Sources/ |
|
70
|
└── Onde/ |
|
71
|
└── onde.swift # UniFFI-generated Swift glue (do NOT edit manually) |
|
72
|
``` |
|
73
|
|
|
74
|
--- |
|
75
|
|
|
76
|
## Architecture |
|
77
|
|
|
78
|
### Layer Diagram |
|
79
|
|
|
80
|
``` |
|
81
|
┌──────────────────────────────────────────────────────────┐ |
|
82
|
│ Swift (iOS / tvOS / macOS) Rust app / CLI / server │ |
|
83
|
│ import Onde use onde::inference::* │ |
|
84
|
└──────────┬───────────────────────────────┬───────────────┘ |
|
85
|
│ UniFFI FFI bindings │ Direct Rust API |
|
86
|
▼ ▼ |
|
87
|
┌──────────────────────────┐ ┌─────────────────────────────┐ |
|
88
|
│ OndeChatEngine │ │ ChatEngine │ |
|
89
|
│ (ffi.rs — uniffi::Object│──│ (engine.rs — Rust-native) │ |
|
90
|
│ FFI-safe, Arc<Self>) │ │ generics, mpsc channels │ |
|
91
|
└──────────┬───────────────┘ └─────────────┬───────────────┘ |
|
92
|
│ │ |
|
93
|
└────────────────┬───────────────┘ |
|
94
|
▼ |
|
95
|
┌────────────────────────┐ |
|
96
|
│ mistralrs::Model │ |
|
97
|
│ GgufModelBuilder │ |
|
98
|
│ Metal / CUDA / CPU │ |
|
99
|
└────────────────────────┘ |
|
100
|
``` |
|
101
|
|
|
102
|
### Key Design Rules |
|
103
|
|
|
104
|
1. **`ChatEngine`** (`engine.rs`) owns all Rust-idiomatic logic: `impl Into<String>`, `tokio::sync::mpsc::Receiver`, etc. Never add UniFFI annotations here. |
|
105
|
2. **`OndeChatEngine`** (`ffi.rs`) is a thin `Arc`-wrapped UniFFI `Object` with concrete, FFI-safe signatures. It delegates every call to the inner `ChatEngine`. |
|
106
|
3. **`StreamChunkListener`** is a `#[uniffi::export(callback_interface)]` trait — it must be passed as a parameter to **free functions** (not Object methods) because UniFFI 0.31 does not support callback_interface on Object methods. |
|
107
|
4. **Free functions** in `ffi.rs` (e.g. `stream_chat_message`, `default_model_config`, `user_message`) are exported with `#[uniffi::export]` and generate top-level Swift functions. |
|
108
|
5. **`uniffi::setup_scaffolding!()`** lives in `lib.rs`. Never call `uniffi::generate_scaffolding!()` from `build.rs` for the proc-macro path. |
|
109
|
|
|
110
|
--- |
|
111
|
|
|
112
|
## Platform Support |
|
113
|
|
|
114
|
| Platform | `target_os` | GPU Backend | Default Model | mistralrs features | |
|
115
|
| -------- | ----------- | ----------------- | ------------- | ------------------ | |
|
116
|
| macOS | `macos` | Metal | Qwen 2.5 Coder 3B (~1.93 GB) | `["metal"]` | |
|
117
|
| iOS | `ios` | Metal | Qwen 2.5 Coder 1.5B (~941 MB) | `["metal"]` | |
|
118
|
| tvOS | `tvos` | Metal | Qwen 2.5 Coder 1.5B (~941 MB) | `["metal"]` | |
|
119
|
| Android | `android` | CPU (candle) | Qwen 2.5 Coder 1.5B (~941 MB) | `[]` + hf-hub | |
|
120
|
| Windows | `windows` | CPU (CUDA in CI) | Qwen 2.5 Coder 3B (~1.93 GB) | `[]` | |
|
121
|
| Linux | `linux` | CPU (CUDA in CI) | Qwen 2.5 Coder 3B (~1.93 GB) | `[]` | |
|
122
|
|
|
123
|
- **Current default routing:** `platform_default()` now prefers the Coder variants, 1.5B on mobile and 3B on desktop. |
|
124
|
- **iOS / tvOS memory constraint:** iOS gives apps about 2–3 GB. The 3B model (~1.93 GB) can still cause OOM on constrained devices, so mobile defaults stay on the 1.5B model. |
|
125
|
- **tvOS tier-3 target:** requires `cargo +nightly -Z build-std`. Stable toolchain cannot build tvOS targets. |
|
126
|
- **Android `hf_hub`:** `dirs::home_dir()` panics in the Android sandbox. `hf-hub` is added as an explicit dependency on Android so `HF_HOME` can be seeded programmatically via `hf_hub::api::tokio::ApiBuilder`. |
|
127
|
|
|
128
|
--- |
|
129
|
|
|
130
|
## Supported Models |
|
131
|
|
|
132
|
All model constants live in `src/inference/models.rs`. When adding a new model: |
|
133
|
|
|
134
|
1. Add `pub const` entries for the HF repo ID, GGUF filename, and, on Android, `TOK_MODEL_ID` when needed. |
|
135
|
2. Add the repo ID to `SUPPORTED_MODELS` so `list_local_hf_models` filters it. |
|
136
|
3. Add a `SupportedModelInfo` entry to `SUPPORTED_MODEL_INFO` with accurate `expected_size_bytes` from the HF API `siblings[].size`. |
|
137
|
4. Add a constructor to `GgufModelConfig` in `engine.rs`. |
|
138
|
5. Export a free function in `ffi.rs` for UniFFI consumers when the model should be reachable from Swift and Kotlin. |
|
139
|
6. If the GGUF does not ship with a built-in chat template, set `chat_template` explicitly. DeepSeek Coder is the current example. |
|
140
|
|
|
141
|
### Current Models |
|
142
|
|
|
143
|
| Model | Repo | File | Size | Platforms | |
|
144
|
|-------|------|------|------|-----------| |
|
145
|
| Qwen 2.5 1.5B Instruct (GGUF Q4_K_M) | `bartowski/Qwen2.5-1.5B-Instruct-GGUF` | `Qwen2.5-1.5B-Instruct-Q4_K_M.gguf` | ~941 MB | All platforms | |
|
146
|
| Qwen 2.5 3B Instruct (GGUF Q4_K_M) | `bartowski/Qwen2.5-3B-Instruct-GGUF` | `Qwen2.5-3B-Instruct-Q4_K_M.gguf` | ~1.93 GB | All platforms | |
|
147
|
| Qwen 2.5 Coder 1.5B Instruct (GGUF Q4_K_M) | `bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF` | `Qwen2.5-Coder-1.5B-Instruct-Q4_K_M.gguf` | ~941 MB | All platforms (mobile default) | |
|
148
|
| Qwen 2.5 Coder 3B Instruct (GGUF Q4_K_M) | `bartowski/Qwen2.5-Coder-3B-Instruct-GGUF` | `Qwen2.5-Coder-3B-Instruct-Q4_K_M.gguf` | ~1.93 GB | All platforms (desktop default) | |
|
149
|
| Qwen 2.5 Coder 7B Instruct (GGUF Q4_K_M) | `bartowski/Qwen2.5-Coder-7B-Instruct-GGUF` | `Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf` | ~4.4 GB | Higher-memory devices | |
|
150
|
| Qwen 3 1.7B (GGUF Q4_K_M) | `bartowski/Qwen3-1.7B-GGUF` | `Qwen3-1.7B-Q4_K_M.gguf` | ~1.3 GB | All platforms | |
|
151
|
| Qwen 3 4B (GGUF Q4_K_M) | `bartowski/Qwen3-4B-GGUF` | `Qwen3-4B-Q4_K_M.gguf` | ~2.7 GB | All platforms | |
|
152
|
| Qwen 3 8B (GGUF Q4_K_M) | `bartowski/Qwen3-8B-GGUF` | `Qwen3-8B-Q4_K_M.gguf` | ~5 GB | Higher-memory devices | |
|
153
|
| Qwen 3 14B (GGUF Q4_K_M) | `bartowski/Qwen3-14B-GGUF` | `Qwen3-14B-Q4_K_M.gguf` | ~8.4 GB | Higher-memory devices | |
|
154
|
| DeepSeek Coder 6.7B Instruct (GGUF Q4_K_M) | `bartowski/deepseek-coder-6.7b-instruct-GGUF` | `deepseek-coder-6.7b-instruct-Q4_K_M.gguf` | ~3.8 GB | Higher-memory devices, custom chat template | |
|
155
|
| Qwen 2.5 Coder 7B Instruct (ISQ) | `Qwen/Qwen2.5-Coder-7B-Instruct` | safetensors (ISQ in-situ) | ~8 GB | macOS (ISQ pipeline) | |
|
156
|
|
|
157
|
--- |
|
158
|
|
|
159
|
## Key Types (Rust ↔ Swift) |
|
160
|
|
|
161
|
| Rust Type | Swift Type | Notes | |
|
162
|
|-----------|------------|-------| |
|
163
|
| `ChatRole` | `enum ChatRole` | `.system`, `.user`, `.assistant` | |
|
164
|
| `ChatMessage` | `struct ChatMessage` | `role: ChatRole`, `content: String` | |
|
165
|
| `SamplingConfig` | `struct SamplingConfig` | All fields `Optional` | |
|
166
|
| `GgufModelConfig` | `struct GgufModelConfig` | `modelId`, `files`, `tokModelId?`, `displayName`, `approxMemory`, `chatTemplate?` | |
|
167
|
| `IsqModelConfig` | `struct IsqModelConfig` | `modelId`, `isqBits: UInt8`, `displayName`, `approxMemory` | |
|
168
|
| `InferenceResult` | `struct InferenceResult` | `text`, `durationSecs`, `durationDisplay`, `finishReason`, `toolCalls` | |
|
169
|
| `ToolCallInfo` | `struct ToolCallInfo` | Structured tool call request emitted by the model | |
|
170
|
| `StreamChunk` | `struct StreamChunk` | `delta`, `done`, `finishReason?` | |
|
171
|
| `EngineStatus` | `enum EngineStatus` | `.unloaded`, `.loading`, `.ready`, `.generating`, `.error` | |
|
172
|
| `EngineInfo` | `struct EngineInfo` | `status`, `modelName?`, `approxMemory?`, `historyLength: UInt64` | |
|
173
|
| `InferenceError` | `enum InferenceError: Error` | `noModelLoaded`, `alreadyLoaded`, `modelBuild`, `inference`, `cancelled`, `other` | |
|
174
|
| `OndeChatEngine` | `class OndeChatEngine` | Thread-safe; `Arc`-backed; constructed with `OndeChatEngine()` | |
|
175
|
| `StreamChunkListener` | `protocol StreamChunkListener` | Implement `onChunk(chunk:) -> Bool` | |
|
176
|
|
|
177
|
--- |
|
178
|
|
|
179
|
## HuggingFace Token |
|
180
|
|
|
181
|
`src/inference/token.rs` resolves the HF token in priority order: |
|
182
|
|
|
183
|
1. **Build-time literal** (`HF_TOKEN` env var baked in via `option_env!`) — required for iOS/tvOS (no filesystem token possible). |
|
184
|
2. **On-disk cache** (`~/.cache/huggingface/token`) — works on macOS after `mistralrs login`. |
|
185
|
|
|
186
|
Set `HF_TOKEN` before building: |
|
187
|
|
|
188
|
```bash |
|
189
|
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
|
190
|
cargo build --release |
|
191
|
``` |
|
192
|
|
|
193
|
On sandboxed platforms, your app's setup function must also configure `HF_HOME`, `HF_HUB_CACHE`, and `TMPDIR` to point inside the app container before any `OndeChatEngine` method is called. See `docs/swift-package.md` for the full `setupInferenceEnvironment()` Swift snippet. |
|
194
|
|
|
195
|
--- |
|
196
|
|
|
197
|
## Building (Rust) |
|
198
|
|
|
199
|
### Prerequisites |
|
200
|
|
|
201
|
```bash |
|
202
|
# Stable toolchain (macOS, iOS, Android, Windows, Linux) |
|
203
|
rustup toolchain install stable |
|
204
|
|
|
205
|
# Nightly toolchain (tvOS tier-3 targets only) |
|
206
|
rustup toolchain install nightly |
|
207
|
rustup component add rust-src --toolchain nightly |
|
208
|
|
|
209
|
# Apple targets (stable) |
|
210
|
rustup target add aarch64-apple-ios aarch64-apple-ios-sim |
|
211
|
rustup target add aarch64-apple-darwin x86_64-apple-darwin |
|
212
|
``` |
|
213
|
|
|
214
|
### Commands |
|
215
|
|
|
216
|
```bash |
|
217
|
# Verify compilation (macOS host) |
|
218
|
cargo check |
|
219
|
|
|
220
|
# Run all tests |
|
221
|
cargo test |
|
222
|
|
|
223
|
# Run inference module tests only |
|
224
|
cargo test inference:: |
|
225
|
|
|
226
|
# Run FFI wrapper tests only |
|
227
|
cargo test inference::ffi:: |
|
228
|
|
|
229
|
# Clippy lint |
|
230
|
cargo clippy |
|
231
|
|
|
232
|
# Format |
|
233
|
cargo fmt |
|
234
|
``` |
|
235
|
|
|
236
|
### tvOS |
|
237
|
|
|
238
|
tvOS targets require nightly and `-Z build-std`: |
|
239
|
|
|
240
|
```bash |
|
241
|
cargo +nightly rustc -Z build-std \ |
|
242
|
--target aarch64-apple-tvos --release --lib --crate-type staticlib |
|
243
|
``` |
|
244
|
|
|
245
|
--- |
|
246
|
|
|
247
|
## Building the Swift XCFramework |
|
248
|
|
|
249
|
The XCFramework bundles staticlibs for iOS device, iOS simulator, tvOS device, tvOS simulator, and macOS into a single distributable that `Package.swift` references. |
|
250
|
|
|
251
|
### Quick build (local or CI) |
|
252
|
|
|
253
|
```bash |
|
254
|
.github/scripts/build-swift-xcframework.sh |
|
255
|
``` |
|
256
|
|
|
257
|
Output lands in `dist/swift/`: |
|
258
|
- `OndeFramework.xcframework/` — the framework tree |
|
259
|
- `OndeFramework.xcframework.zip` — zipped for SPM remote binary |
|
260
|
- `OndeFramework.checksum.txt` — SHA-256 for `Package.swift` |
|
261
|
- `version.txt` — semver from `Cargo.toml` |
|
262
|
- `Package/Sources/Onde/onde.swift` — generated UniFFI Swift glue |
|
263
|
|
|
264
|
### Manual steps |
|
265
|
|
|
266
|
```bash |
|
267
|
# 1. Build the uniffi-bindgen binary (pinned to uniffi =0.31.0) |
|
268
|
cargo build --manifest-path uniffi-bindgen/Cargo.toml --release |
|
269
|
BINDGEN=uniffi-bindgen/target/release/uniffi-bindgen |
|
270
|
|
|
271
|
# 2. Compile staticlibs per target (use +1.92.0 or current stable) |
|
272
|
cargo +stable rustc --target aarch64-apple-ios --release --lib --crate-type staticlib |
|
273
|
cargo +stable rustc --target aarch64-apple-ios-sim --release --lib --crate-type staticlib |
|
274
|
cargo +stable rustc --target aarch64-apple-darwin --release --lib --crate-type staticlib |
|
275
|
cargo +nightly rustc -Z build-std --target aarch64-apple-tvos --release --lib --crate-type staticlib |
|
276
|
cargo +nightly rustc -Z build-std --target aarch64-apple-tvos-sim --release --lib --crate-type staticlib |
|
277
|
|
|
278
|
# 3. Generate Swift bindings from the iOS arm64 slice |
|
279
|
$BINDGEN generate target/aarch64-apple-ios/release/libonde.a \ |
|
280
|
--language swift \ |
|
281
|
--out-dir sdk/Onde/Sources/Onde \ |
|
282
|
--config uniffi.toml |
|
283
|
|
|
284
|
# 4. Assemble the XCFramework (see build-swift-xcframework.sh for full xcodebuild invocation) |
|
285
|
``` |
|
286
|
|
|
287
|
### Updating `onde-swift` after a new release |
|
288
|
|
|
289
|
This is fully automated by CI. Publishing a GitHub Release on `onde` triggers the workflow, which: |
|
290
|
|
|
291
|
1. Builds the XCFramework and attaches it to the release. |
|
292
|
2. Rewrites `onde-swift/Package.swift` with the release `url:` + `checksum:`. |
|
293
|
3. Copies the freshly generated `onde.swift` into `onde-swift/Sources/Onde/`. |
|
294
|
4. Commits, tags, and pushes `onde-swift` — Swift Package Index picks it up automatically. |
|
295
|
|
|
296
|
Manual intervention is only needed if `ONDE_SWIFT_PAT` has expired or the `onde-swift` push fails. |
|
297
|
|
|
298
|
--- |
|
299
|
|
|
300
|
## Swift SDK (`onde-swift`) |
|
301
|
|
|
302
|
### Package.swift structure |
|
303
|
|
|
304
|
``` |
|
305
|
onde-swift/ |
|
306
|
├── Package.swift # Declares OndeFramework binary target + Onde wrapper target |
|
307
|
└── Sources/Onde/ |
|
308
|
└── onde.swift # UniFFI-generated glue (NEVER edit manually) |
|
309
|
``` |
|
310
|
|
|
311
|
`Package.swift` has two targets: |
|
312
|
|
|
313
|
- **`OndeFramework`** — `.binaryTarget` using `url:` + `checksum:` for distribution. For local development, swap to the `path:` form documented in the `Package.swift` header comment and run `build-swift-xcframework.sh` first. |
|
314
|
- **`Onde`** — `.target` depending on `OndeFramework`, used as the public import name in Swift. |
|
315
|
|
|
316
|
### Swift API Quick Reference |
|
317
|
|
|
318
|
```swift |
|
319
|
import Onde |
|
320
|
|
|
321
|
// Create engine |
|
322
|
let engine = OndeChatEngine() |
|
323
|
|
|
324
|
// Load the platform default model |
|
325
|
let elapsed = try await engine.loadDefaultModel( |
|
326
|
systemPrompt: "You are a helpful assistant.", |
|
327
|
sampling: nil |
|
328
|
) |
|
329
|
|
|
330
|
// Or load the model assigned to your app in the Onde dashboard |
|
331
|
let assignedElapsed = try await engine.loadAssignedModel( |
|
332
|
appId: "your-app-id", |
|
333
|
appSecret: "your-app-secret", |
|
334
|
systemPrompt: "You are a helpful assistant.", |
|
335
|
sampling: nil |
|
336
|
) |
|
337
|
|
|
338
|
// Multi-turn chat |
|
339
|
let result = try await engine.sendMessage(message: "Hello!") |
|
340
|
print(result.text) |
|
341
|
print(result.toolCalls) |
|
342
|
|
|
343
|
// Streaming (free function — callback_interface constraint in UniFFI 0.31) |
|
344
|
class Handler: StreamChunkListener { |
|
345
|
func onChunk(chunk: StreamChunk) -> Bool { |
|
346
|
print(chunk.delta, terminator: "") |
|
347
|
return !chunk.done |
|
348
|
} |
|
349
|
} |
|
350
|
try await streamChatMessage(engine: engine, message: "Tell me a story.", listener: Handler()) |
|
351
|
|
|
352
|
// One-shot (does NOT modify conversation history) |
|
353
|
let enhanced = try await engine.generate( |
|
354
|
messages: [userMessage(content: "Expand: a cat in space")], |
|
355
|
sampling: deterministicSamplingConfig() |
|
356
|
) |
|
357
|
|
|
358
|
// Status |
|
359
|
let info = await engine.info() // EngineInfo |
|
360
|
|
|
361
|
// History management |
|
362
|
let history = await engine.history() |
|
363
|
let removed = await engine.clearHistory() |
|
364
|
await engine.pushHistory(message: userMessage(content: "...")) |
|
365
|
|
|
366
|
// Cleanup |
|
367
|
await engine.unloadModel() |
|
368
|
``` |
|
369
|
|
|
370
|
### Free Functions |
|
371
|
|
|
372
|
| Function | Returns | Notes | |
|
373
|
|----------|---------|-------| |
|
374
|
| `defaultModelConfig()` | `GgufModelConfig` | Platform-aware Coder default (1.5B on iOS/tvOS/Android, 3B on desktop) | |
|
375
|
| `qwen251_5bConfig()` | `GgufModelConfig` | Forces Qwen 2.5 1.5B regardless of platform | |
|
376
|
| `qwen253bConfig()` | `GgufModelConfig` | Forces Qwen 2.5 3B regardless of platform | |
|
377
|
| `defaultSamplingConfig()` | `SamplingConfig` | temp=0.7, top_p=0.95, max_tokens=512 | |
|
378
|
| `deterministicSamplingConfig()` | `SamplingConfig` | temp=0.0, greedy | |
|
379
|
| `mobileSamplingConfig()` | `SamplingConfig` | temp=0.7, max_tokens=128 | |
|
380
|
| `systemMessage(content:)` | `ChatMessage` | `.system` role | |
|
381
|
| `userMessage(content:)` | `ChatMessage` | `.user` role | |
|
382
|
| `assistantMessage(content:)` | `ChatMessage` | `.assistant` role | |
|
383
|
| `streamChatMessage(engine:message:listener:)` | `async throws` | Streaming via callback | |
|
384
|
|
|
385
|
--- |
|
386
|
|
|
387
|
## UniFFI Conventions |
|
388
|
|
|
389
|
- **Version pin:** `uniffi = "=0.31.0"` everywhere — the `onde` crate, `uniffi-bindgen/`, and `[build-dependencies]` must all use the **same** version. Mixing versions causes bindgen panics. |
|
390
|
- **`uniffi::setup_scaffolding!()`** in `lib.rs` — proc-macro approach, no UDL file needed. |
|
391
|
- **Object methods** use `#[uniffi::export]` on the `impl OndeChatEngine` block. |
|
392
|
- **Callback interfaces** (`StreamChunkListener`) must be parameters of **free functions** only, not Object methods. |
|
393
|
- **`Arc<Self>`** is the return type for `#[uniffi::constructor]`. UniFFI automatically handles this. |
|
394
|
- **Async:** all async exported methods use `tokio` runtime (enabled via `uniffi = { features = ["tokio"] }`). |
|
395
|
- **`uniffi.toml`**: lives at the crate root. Adjust renaming or namespace settings there before regenerating. |
|
396
|
|
|
397
|
--- |
|
398
|
|
|
399
|
## HuggingFace Cache (`hf_cache.rs`) |
|
400
|
|
|
401
|
The `hf_cache` module manages the on-device model cache, exposed via UniFFI to Swift / Kotlin as-needed. |
|
402
|
|
|
403
|
Key public functions: |
|
404
|
|
|
405
|
| Function | Description | |
|
406
|
|----------|-------------| |
|
407
|
| `list_local_hf_models()` | List downloaded models that Onde supports | |
|
408
|
| `list_supported_hf_models()` | All supported models with download status | |
|
409
|
| `download_model(model_id, progress_callback)` | Download a model with progress reporting | |
|
410
|
| `delete_local_hf_model(model_id)` | Remove a model from the local cache | |
|
411
|
| `diagnose_hf_cache()` | Inspect the cache for corruption | |
|
412
|
| `repair_hf_cache_symlinks()` | Fix broken symlinks in the HF cache layout | |
|
413
|
| `model_cache_path(model_id)` | Resolve the filesystem path for a model | |
|
414
|
| `clean_stale_lock_files()` | Remove leftover `.lock` files from interrupted downloads | |
|
415
|
|
|
416
|
`ModelDownloadProgress` carries `downloaded_bytes`, `total_bytes`, `progress` (0.0–1.0), and `done`. |
|
417
|
|
|
418
|
--- |
|
419
|
|
|
420
|
## Sampling Presets |
|
421
|
|
|
422
|
| Preset | `temperature` | `top_p` | `max_tokens` | Use Case | |
|
423
|
|--------|--------------|---------|--------------|----------| |
|
424
|
| `SamplingConfig::default()` | 0.7 | 0.95 | 512 | General creative chat | |
|
425
|
| `SamplingConfig::deterministic()` | 0.0 | — | 512 | Reproducible / coding | |
|
426
|
| `SamplingConfig::mobile()` | 0.7 | 0.95 | 128 | Memory/latency constrained | |
|
427
|
| `SamplingConfig::coding()` | 0.0 | — | 512 | Code generation | |
|
428
|
| `SamplingConfig::coding_mobile()` | 0.0 | — | 128 | Code on mobile | |
|
429
|
|
|
430
|
--- |
|
431
|
|
|
432
|
## Testing |
|
433
|
|
|
434
|
```bash |
|
435
|
# All tests (requires a macOS host with Metal) |
|
436
|
cargo test |
|
437
|
|
|
438
|
# Unit tests only (no model downloads) |
|
439
|
cargo test inference:: |
|
440
|
cargo test inference::ffi:: |
|
441
|
cargo test inference::types:: |
|
442
|
|
|
443
|
# hf_cache module |
|
444
|
cargo test hf_cache:: |
|
445
|
|
|
446
|
# Clippy (treat warnings as errors) |
|
447
|
cargo clippy -- -D warnings |
|
448
|
``` |
|
449
|
|
|
450
|
Tests that require model downloads are integration tests and not run by default in CI. Unit tests verify: |
|
451
|
- Type constructors and `Display` implementations |
|
452
|
- `SamplingConfig` presets |
|
453
|
- `GgufModelConfig` constructors and `platform_default()` routing |
|
454
|
- `OndeChatEngine` lifecycle: `new()` starts unloaded, `send_message` without model returns `InferenceError::NoModelLoaded`, `clear_history` on empty returns 0, `unload_model` on empty returns `nil`. |
|
455
|
|
|
456
|
--- |
|
457
|
|
|
458
|
## Known Issues |
|
459
|
|
|
460
|
### `___chkstk_darwin` linker error on tvOS |
|
461
|
|
|
462
|
`aws-lc-sys` (transitive via `reqwest → rustls → aws-lc-rs`) references `___chkstk_darwin`, a stack probing symbol that tvOS does not export. `build.rs` compiles `scripts/tvos_chkstk.s` (a no-op `ret` stub) via the `cc` crate to satisfy the linker. **Do not delete `scripts/tvos_chkstk.s`.** |
|
463
|
|
|
464
|
Affects: tvOS only. macOS, iOS, Android, Windows, Linux are unaffected. |
|
465
|
|
|
466
|
### Metal Toolchain missing (Xcode 26+) |
|
467
|
|
|
468
|
On Xcode 26+, the Metal compiler is a separate download. If you see empty `.metallib` files (92 bytes) or `"Error while loading function: fused_glu_float"` at runtime: |
|
469
|
|
|
470
|
```bash |
|
471
|
xcodebuild -downloadComponent MetalToolchain |
|
472
|
cargo clean -p mistralrs-quant |
|
473
|
cargo check |
|
474
|
``` |
|
475
|
|
|
476
|
### Android `home_dir` sandbox panic |
|
477
|
|
|
478
|
`dirs::home_dir()` panics in the Android sandbox. The crate-level `Cargo.toml` adds `hf-hub` as an explicit Android dependency so `HF_HOME` can be seeded via `ApiBuilder` before any hub request. Never call `home::home_dir()` or `dirs::home_dir()` on Android. |
|
479
|
|
|
480
|
--- |
|
481
|
|
|
482
|
## Code Conventions |
|
483
|
|
|
484
|
### Rust |
|
485
|
|
|
486
|
- **Error handling:** `anyhow` for application-level errors inside `engine.rs`; `thiserror` for the `InferenceError` enum (exported via UniFFI). Never `.unwrap()` or `.expect()` in non-test code. |
|
487
|
- **Async:** `tokio` runtime. All async functions in `OndeChatEngine` are `pub async fn`. |
|
488
|
- **Logging:** `log` crate macros (`log::debug!`, `log::info!`, `log::warn!`, `log::error!`). No `println!` in library code. |
|
489
|
- **Platform gating:** use `#[cfg(target_os = "...")]` blocks. Match `Cargo.toml`'s target-conditional dependency sections. |
|
490
|
- **Re-exports:** `mistralrs`, `hf_hub`, and `mistralrs_core` are re-exported from `lib.rs` for downstream Rust consumers. Keep these re-exports in sync with what's actually available per platform. |
|
491
|
- **No `mod.rs`:** use named files (`inference/engine.rs`) not `inference/mod.rs` — except that `inference/mod.rs` exists and is the intentional module root for the `inference` module. |
|
492
|
|
|
493
|
### Swift / SDK |
|
494
|
|
|
495
|
- **Never manually edit `onde-swift/Sources/Onde/onde.swift`** — it is generated by `uniffi-bindgen`. Regenerate by running the build script. |
|
496
|
- **`Package.swift` binary target** uses `url:` + `checksum:` in the committed form. For local development, swap to `path:` (instructions are in the file header). Never commit the `path:` form — CI overwrites it on every release via `release-sdk-swift.yml`. |
|
497
|
- **iOS/tvOS sandbox:** always call `setupInferenceEnvironment()` at app launch before any `OndeChatEngine` call. |
|
498
|
|
|
499
|
--- |
|
500
|
|
|
501
|
## CI / Release Workflow |
|
502
|
|
|
503
|
### `release-sdk-swift.yml` |
|
504
|
|
|
505
|
Triggered on **semver tag push** or `workflow_dispatch`. Runs on `macos-15`. |
|
506
|
|
|
507
|
Steps: |
|
508
|
1. Install stable Rust with iOS/macOS targets. |
|
509
|
2. Install nightly Rust with `rust-src` for tvOS `-Z build-std`. |
|
510
|
3. Run `build-swift-xcframework.sh`. |
|
511
|
4. Read `version.txt` and `OndeFramework.checksum.txt` into step outputs. |
|
512
|
5. Validate that the tag matches `Cargo.toml`. |
|
513
|
6. Upload CI artifacts: zip, checksum, version, generated `onde.swift`. |
|
514
|
7. On tag push, create a GitHub Release with the zip and checksum attached. |
|
515
|
8. On tag push, check out `ondeinference/onde-swift` using `ONDE_SWIFT_PAT`. |
|
516
|
9. On tag push, rewrite the `.binaryTarget` in `onde-swift/Package.swift` with the release URL and checksum. |
|
517
|
10. On tag push, copy the generated `onde.swift` into `onde-swift/Sources/Onde/`. |
|
518
|
11. On tag push, commit both files, tag the commit with the version, and push to `ondeinference/onde-swift`. |
|
519
|
|
|
520
|
#### Required secret |
|
521
|
|
|
522
|
`ONDE_SWIFT_PAT` — a GitHub Personal Access Token with **`contents: write`** scope on the `ondeinference/onde-swift` repository. |
|
523
|
|
|
524
|
### `release-sdk-kotlin.yml` |
|
525
|
|
|
526
|
Triggered on **semver tag push** or `workflow_dispatch`. Runs on `macos-15` because it builds both Android artifacts and the macOS Apple Silicon JVM native library. |
|
527
|
|
|
528
|
Steps: |
|
529
|
1. Install stable Rust and the `aarch64-apple-darwin` target. |
|
530
|
2. Install Android SDK / NDK and auto-discover the latest installed 29.x NDK for `cargo-ndk`. |
|
531
|
3. Build Android JNI libraries for all supported ABIs. |
|
532
|
4. Build the macOS Apple Silicon `libonde.dylib` used by the JVM target. |
|
533
|
5. Run Gradle publication tasks for the Kotlin Multiplatform package. |
|
534
|
6. Validate that the tag matches `sdk/kotlin/gradle.properties` `VERSION_NAME`. |
|
535
|
7. On tag push, publish `com.ondeinference:onde-inference` to Maven Central. |
|
536
|
|
|
537
|
#### Required secrets |
|
538
|
|
|
539
|
- `ORG_GRADLE_PROJECT_MAVENCENTRALUSERNAME` |
|
540
|
- `ORG_GRADLE_PROJECT_MAVENCENTRALPASSWORD` |
|
541
|
- `ORG_GRADLE_PROJECT_SIGNINGKEYID` |
|
542
|
- `ORG_GRADLE_PROJECT_SIGNINGKEY` |
|
543
|
- `ORG_GRADLE_PROJECT_SIGNINGPASSWORD` |
|
544
|
|
|
545
|
### `release-sdk-dart.yml` |
|
546
|
|
|
547
|
Triggered on **semver tag push** or `workflow_dispatch`. |
|
548
|
|
|
549
|
- Publishes `onde_inference` to pub.dev. |
|
550
|
- Version must match the Dart package version. |
|
551
|
- The example app is part of release validation. |
|
552
|
|
|
553
|
### `release-sdk-npm.yml` |
|
554
|
|
|
555
|
Triggered on **semver tag push** or `workflow_dispatch`. |
|
556
|
|
|
557
|
- Publishes `@ondeinference/react-native` to npm. |
|
558
|
- Builds the Expo module package and validates the published version. |
|
559
|
|
|
560
|
### `release-sdk-rust.yml` |
|
561
|
|
|
562
|
Triggered on **semver tag push** or `workflow_dispatch`. |
|
563
|
|
|
564
|
- Publishes `onde` to crates.io. |
|
565
|
- Uses registry dependencies, not git refs, because `cargo publish` strips git patches. |
|
566
|
|
|
567
|
### Release Process (end-to-end) |
|
568
|
|
|
569
|
1. Bump versions in every affected package manifest: |
|
570
|
- `Cargo.toml` |
|
571
|
- `sdk/kotlin/gradle.properties` |
|
572
|
- `sdk/dart/pubspec.yaml` |
|
573
|
- `sdk/react-native/package.json` |
|
574
|
2. Merge release branches with `--no-ff`. Do not fast-forward multi-commit feature branches. |
|
575
|
3. Commit, tag, and push the release. |
|
576
|
4. CI publishes all five SDKs: |
|
577
|
- crates.io (`onde`) |
|
578
|
- Swift Package Index / GitHub Releases (`onde-swift`) |
|
579
|
- Maven Central (`com.ondeinference:onde-inference`) |
|
580
|
- pub.dev (`onde_inference`) |
|
581
|
- npm (`@ondeinference/react-native`) |
|
582
|
5. Verify the registries after publish, especially Maven Central indexing and the generated Swift release assets. |
|
583
|
|
|
584
|
> **Note:** `onde-swift/Package.swift` is updated automatically by CI. For local development only, switch it to the `path:` form documented in the file header, then switch it back before committing. |
|
585
|
|
|
586
|
--- |
|
587
|
|
|
588
|
## Distribution Registry Summary |
|
589
|
|
|
590
|
| Registry | Name | Import | |
|
591
|
|----------|------|--------| |
|
592
|
| crates.io | `onde` | `onde = "1.x"` | |
|
593
|
| Swift Package Index | `onde-swift` (org: `ondeinference`) | `import Onde` | |
|
594
|
| Maven Central | `com.ondeinference:onde-inference` | Gradle `implementation("com.ondeinference:onde-inference:<version>")` | |
|
595
|
| pub.dev | `onde_inference` | `import 'package:onde_inference/onde_inference.dart';` | |
|
596
|
| npm | `@ondeinference/react-native` | `import { OndeChatEngine } from "@ondeinference/react-native"` | |
|
597
|
| PyPI | `onde-inference` | `import onde_inference` | |
|
598
|
| RubyGems | `onde-inference` | `require 'onde'` | |
|
599
|
|
|
600
|
--- |
|
601
|
|
|
602
|
*This file is for AI agent and human developer reference. Update it when the architecture, API surface, supported models, or toolchain requirements change.* |
|
603
|
``` |