|
1
|
<p align="center"> |
|
2
|
<img src="https://raw.githubusercontent.com/ondeinference/onde/main/assets/onde-inference-logo.svg" alt="Onde Inference" width="96"> |
|
3
|
</p> |
|
4
|
|
|
5
|
<h1 align="center">Onde Inference</h1> |
|
6
|
|
|
7
|
<p align="center"> |
|
8
|
<strong>Run LLMs on-device from Flutter and Dart with <a href="https://ondeinference.com/">Onde Inference</a>. Metal on iOS and macOS, CPU everywhere else.</strong> |
|
9
|
</p> |
|
10
|
|
|
11
|
<p align="center"> |
|
12
|
<a href="https://pub.dev/packages/onde_inference"><img src="https://img.shields.io/pub/v/onde_inference?style=flat-square&color=235843&labelColor=17211D&label=pub.dev" alt="pub.dev"></a> |
|
13
|
<a href="https://ondeinference.com"><img src="https://img.shields.io/badge/ondeinference.com-235843?style=flat-square&labelColor=17211D" alt="Website"></a> |
|
14
|
<a href="https://github.com/ondeinference/onde/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-235843?style=flat-square&labelColor=17211D" alt="License"></a> |
|
15
|
</p> |
|
16
|
|
|
17
|
<p align="center"> |
|
18
|
<a href="https://github.com/ondeinference/onde">Rust SDK</a> · <a href="https://swiftpackageindex.com/ondeinference/onde-swift">Swift SDK</a> · <a href="https://central.sonatype.com/artifact/com.ondeinference/onde-inference">Kotlin Multiplatform SDK</a> · <a href="https://www.npmjs.com/package/@ondeinference/react-native">React Native SDK</a> · <a href="https://ondeinference.com">Website</a> |
|
19
|
</p> |
|
20
|
|
|
21
|
--- |
|
22
|
|
|
23
|
Run Qwen 2.5 models directly inside your Flutter app. The model downloads from Hugging Face on first launch, then everything runs locally. No server, no API key, and no user data leaves the device. On an iPhone 15 Pro, Metal reaches around 15 tok/s. Android, Linux, and Windows run on CPU, so they are slower but still useful for fully local inference. |
|
24
|
|
|
25
|
You get multi-turn chat, streaming, one-shot generation, configurable sampling, and structured tool call metadata in one package. |
|
26
|
|
|
27
|
## Platform support |
|
28
|
|
|
29
|
| Platform | Backend | Default model | Notes | |
|
30
|
|----------|---------|---------------|-------| |
|
31
|
| iOS 13+ | Metal | Qwen 2.5 Coder 1.5B (~941 MB) | CocoaPods and Swift Package Manager plugin manifests are included | |
|
32
|
| macOS 10.15+ | Metal | Qwen 2.5 Coder 3B (~1.93 GB) | CocoaPods and Swift Package Manager plugin manifests are included | |
|
33
|
| Android API 21+ | CPU | Qwen 2.5 Coder 1.5B (~941 MB) | arm64-v8a, armeabi-v7a, x86_64 by default; see [Android ABIs](#android-abis) | |
|
34
|
| Linux x86_64 | CPU | Qwen 2.5 Coder 3B (~1.93 GB) | CUDA possible, see docs | |
|
35
|
| Windows x86_64 | CPU | Qwen 2.5 Coder 3B (~1.93 GB) | CUDA possible, see docs | |
|
36
|
|
|
37
|
Web is not supported. On-device inference needs native system access that browsers do not expose. |
|
38
|
|
|
39
|
--- |
|
40
|
|
|
41
|
## Quick start |
|
42
|
|
|
43
|
```yaml |
|
44
|
dependencies: |
|
45
|
onde_inference: ^1.0.2 |
|
46
|
``` |
|
47
|
|
|
48
|
The inference engine is written in Rust and connected to Dart through [flutter_rust_bridge](https://pub.dev/packages/flutter_rust_bridge). You need a working [Rust toolchain](https://rustup.rs). The first build is usually slow because it compiles the full native dependency tree. |
|
49
|
|
|
50
|
### Initialize |
|
51
|
|
|
52
|
Call this once at startup before creating any `OndeChatEngine`: |
|
53
|
|
|
54
|
```dart |
|
55
|
import 'package:flutter/widgets.dart'; |
|
56
|
import 'package:onde_inference/onde_inference.dart'; |
|
57
|
|
|
58
|
Future<void> main() async { |
|
59
|
WidgetsFlutterBinding.ensureInitialized(); |
|
60
|
await OndeInference.init(); |
|
61
|
runApp(const MyApp()); |
|
62
|
} |
|
63
|
``` |
|
64
|
|
|
65
|
### Load a model |
|
66
|
|
|
67
|
```dart |
|
68
|
final engine = OndeChatEngine(); |
|
69
|
|
|
70
|
final elapsed = await engine.loadDefaultModel( |
|
71
|
systemPrompt: 'You are a helpful assistant.', |
|
72
|
); |
|
73
|
|
|
74
|
print('Model loaded in ${elapsed.toStringAsFixed(1)} s'); |
|
75
|
``` |
|
76
|
|
|
77
|
For production, you can load the model assigned to your Onde app from the dashboard: |
|
78
|
|
|
79
|
```dart |
|
80
|
final assignedElapsed = await engine.loadAssignedModel( |
|
81
|
appId: 'your-app-id', |
|
82
|
appSecret: 'your-app-secret', |
|
83
|
systemPrompt: 'You are a helpful assistant.', |
|
84
|
); |
|
85
|
|
|
86
|
print('Assigned model loaded in ${assignedElapsed.toStringAsFixed(1)} s'); |
|
87
|
``` |
|
88
|
|
|
89
|
### Chat |
|
90
|
|
|
91
|
```dart |
|
92
|
final result = await engine.sendMessage( |
|
93
|
message: 'What is Rust ownership?', |
|
94
|
); |
|
95
|
|
|
96
|
print(result.text); |
|
97
|
print(result.durationDisplay); |
|
98
|
print(result.toolCalls); |
|
99
|
``` |
|
100
|
|
|
101
|
### Stream |
|
102
|
|
|
103
|
```dart |
|
104
|
final buffer = StringBuffer(); |
|
105
|
|
|
106
|
await for (final chunk in engine.streamMessage(message: 'Tell me a short story.')) { |
|
107
|
buffer.write(chunk.delta); |
|
108
|
if (chunk.done) break; |
|
109
|
} |
|
110
|
|
|
111
|
print(buffer.toString()); |
|
112
|
``` |
|
113
|
|
|
114
|
### Status and history |
|
115
|
|
|
116
|
```dart |
|
117
|
final info = await engine.info(); |
|
118
|
print(info.status); |
|
119
|
print(info.modelName); |
|
120
|
print(info.approxMemory); |
|
121
|
print(info.historyLength); |
|
122
|
|
|
123
|
final history = await engine.history(); |
|
124
|
for (final msg in history) { |
|
125
|
print('${msg.role}: ${msg.content}'); |
|
126
|
} |
|
127
|
|
|
128
|
final removed = await engine.clearHistoryCount(); |
|
129
|
print('Cleared $removed messages.'); |
|
130
|
``` |
|
131
|
|
|
132
|
### One-shot generation |
|
133
|
|
|
134
|
This runs inference without modifying conversation history. |
|
135
|
|
|
136
|
```dart |
|
137
|
final result = await engine.generate( |
|
138
|
messages: [ |
|
139
|
ChatMessage(role: ChatRole.system, content: 'Output valid JSON only.'), |
|
140
|
ChatMessage(role: ChatRole.user, content: 'Name: Alice, Age: 30'), |
|
141
|
], |
|
142
|
sampling: OndeInference.deterministicSamplingConfig(), |
|
143
|
); |
|
144
|
|
|
145
|
print(result.text); |
|
146
|
``` |
|
147
|
|
|
148
|
### Unload |
|
149
|
|
|
150
|
```dart |
|
151
|
await engine.unloadModel(); |
|
152
|
``` |
|
153
|
|
|
154
|
--- |
|
155
|
|
|
156
|
## Model selection |
|
157
|
|
|
158
|
```dart |
|
159
|
final config = OndeInference.defaultModelConfig(); |
|
160
|
final small = OndeInference.qwen2515bConfig(); |
|
161
|
final medium = OndeInference.qwen253bConfig(); |
|
162
|
final coder = OndeInference.qwen25Coder3bConfig(); |
|
163
|
|
|
164
|
await engine.loadGgufModel( |
|
165
|
config: coder, |
|
166
|
systemPrompt: 'You are an expert software engineer.', |
|
167
|
); |
|
168
|
``` |
|
169
|
|
|
170
|
Pre-quantized UQFF models use the base repository (or local model directory) |
|
171
|
for tokenizer/configuration resolution and the first UQFF shard as the file: |
|
172
|
|
|
173
|
```dart |
|
174
|
final uqff = OndeInference.uqffModelConfig( |
|
175
|
modelId: 'google/gemma-4-E4B-it', |
|
176
|
files: ['q4k-0.uqff'], |
|
177
|
displayName: 'Gemma 4 E4B (UQFF Q4K)', |
|
178
|
approxMemory: '~2.5 GB (UQFF Q4K)', |
|
179
|
); |
|
180
|
|
|
181
|
await engine.loadUqffModel( |
|
182
|
config: uqff, |
|
183
|
systemPrompt: 'You are an expert software engineer.', |
|
184
|
); |
|
185
|
``` |
|
186
|
|
|
187
|
For sharded UQFF models, pass the first shard; mistral.rs discovers sibling |
|
188
|
shards with the same prefix. |
|
189
|
|
|
190
|
| Model | Size | Good for | |
|
191
|
|-------|------|----------| |
|
192
|
| Qwen 2.5 1.5B Instruct Q4_K_M | ~941 MB | iOS, tvOS, Android | |
|
193
|
| Qwen 2.5 3B Instruct Q4_K_M | ~1.93 GB | macOS, Linux, Windows | |
|
194
|
| Qwen 2.5 Coder 1.5B Instruct Q4_K_M | ~941 MB | Code on mobile | |
|
195
|
| Qwen 2.5 Coder 3B Instruct Q4_K_M | ~1.93 GB | Code on desktop | |
|
196
|
|
|
197
|
--- |
|
198
|
|
|
199
|
## Sampling |
|
200
|
|
|
201
|
All sampling fields are optional. `null` means "use the engine default". |
|
202
|
|
|
203
|
```dart |
|
204
|
final sampling = SamplingConfig( |
|
205
|
temperature: 0.7, |
|
206
|
topP: 0.95, |
|
207
|
topK: BigInt.from(40), |
|
208
|
maxTokens: BigInt.from(256), |
|
209
|
); |
|
210
|
|
|
211
|
await engine.setSampling(sampling: sampling); |
|
212
|
``` |
|
213
|
|
|
214
|
Presets: |
|
215
|
|
|
216
|
```dart |
|
217
|
OndeInference.defaultSamplingConfig(); |
|
218
|
OndeInference.deterministicSamplingConfig(); |
|
219
|
OndeInference.mobileSamplingConfig(); |
|
220
|
``` |
|
221
|
|
|
222
|
--- |
|
223
|
|
|
224
|
## Error handling |
|
225
|
|
|
226
|
The generated bridge throws `OndeError` values directly: |
|
227
|
|
|
228
|
```dart |
|
229
|
try { |
|
230
|
await engine.loadDefaultModel(); |
|
231
|
} on OndeError catch (e) { |
|
232
|
debugPrint('Inference error: $e'); |
|
233
|
} |
|
234
|
``` |
|
235
|
|
|
236
|
Common causes include calling `sendMessage` before loading a model, having no internet on first run while the model still needs to download, or running out of memory. |
|
237
|
|
|
238
|
--- |
|
239
|
|
|
240
|
## Sandboxed app setup (iOS / macOS / Android) |
|
241
|
|
|
242
|
On iOS, macOS, and Android, configure the Hugging Face cache directory before loading a model. On Apple platforms, Onde first tries the shared App Group container (`group.com.ondeinference.apps`) and falls back to your provided directory. |
|
243
|
|
|
244
|
```dart |
|
245
|
import 'dart:io' show Platform; |
|
246
|
|
|
247
|
import 'package:flutter/widgets.dart'; |
|
248
|
import 'package:onde_inference/onde_inference.dart'; |
|
249
|
import 'package:path_provider/path_provider.dart'; |
|
250
|
|
|
251
|
Future<void> main() async { |
|
252
|
WidgetsFlutterBinding.ensureInitialized(); |
|
253
|
await OndeInference.init(); |
|
254
|
|
|
255
|
String? fallbackDir; |
|
256
|
if (Platform.isIOS || Platform.isAndroid) { |
|
257
|
final dir = await getApplicationSupportDirectory(); |
|
258
|
fallbackDir = dir.path; |
|
259
|
} |
|
260
|
|
|
261
|
await OndeInference.setupCacheDir(fallbackDir: fallbackDir); |
|
262
|
runApp(const MyApp()); |
|
263
|
} |
|
264
|
``` |
|
265
|
|
|
266
|
--- |
|
267
|
|
|
268
|
## Android ABIs |
|
269
|
|
|
270
|
The Rust engine is cross-compiled during `flutter build apk` by the plugin's |
|
271
|
Gradle module, which shells out to [`cargo-ndk`](https://github.com/bbqsrc/cargo-ndk). |
|
272
|
You need it once per machine, along with the Rust targets: |
|
273
|
|
|
274
|
```bash |
|
275
|
cargo install cargo-ndk |
|
276
|
rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android |
|
277
|
``` |
|
278
|
|
|
279
|
The NDK comes from whichever version Gradle already resolved, so no |
|
280
|
`ANDROID_NDK_HOME` export is needed for Gradle-driven builds. |
|
281
|
|
|
282
|
Unless you say otherwise, all three default ABIs are built: arm64-v8a, |
|
283
|
armeabi-v7a, and x86_64. Each is a full release build of a large crate, so this |
|
284
|
is slow. While iterating, build only the one your device needs: |
|
285
|
|
|
286
|
```bash |
|
287
|
flutter run -d <device> --android-project-arg=onde.androidAbis=arm64-v8a |
|
288
|
``` |
|
289
|
|
|
290
|
## Example app |
|
291
|
|
|
292
|
A full Flutter example lives in `example/`. It demonstrates: |
|
293
|
|
|
294
|
- `OndeChatEngine()` lifecycle |
|
295
|
- assigned-model loading with dashboard credentials |
|
296
|
- streaming chat UI |
|
297
|
- sampling preset switching |
|
298
|
- cache directory setup for sandboxed platforms |
|
299
|
|
|
300
|
Run it locally from `sdk/dart/example/`. |
|
301
|
|
|
302
|
## Contributing |
|
303
|
|
|
304
|
The source lives at [github.com/ondeinference/onde](https://github.com/ondeinference/onde): |
|
305
|
|
|
306
|
- Rust core: `src/` |
|
307
|
- Dart bridge crate: `sdk/dart/rust/` |
|
308
|
- Dart library: `sdk/dart/lib/` |
|
309
|
- Example app: `sdk/dart/example/` |
|
310
|
|
|
311
|
Open an issue before sending large PRs. |
|
312
|
|
|
313
|
## License |
|
314
|
|
|
315
|
Onde is dual-licensed under **MIT** and **Apache 2.0**. You can use either one. |
|
316
|
|
|
317
|
- [MIT License](https://github.com/ondeinference/onde/blob/main/LICENSE-MIT) |
|
318
|
- [Apache License 2.0](https://github.com/ondeinference/onde/blob/main/LICENSE-APACHE) |
|
319
|
|
|
320
|
--- |
|
321
|
|
|
322
|
## Copyright |
|
323
|
|
|
324
|
© 2026 [Splitfire AB](https://5mb.app) ([Onde Inference](https://ondeinference.com)). |