@ondeinference / onde
Code Commits Issues Pulls Cloud Agent CI/CD
copilot/assess-inference-engine-complexity
copilot/assess-inference-engine-complexity copilot/fix-onde-npm-sdk-v1-1-2-release development feature/diffusion-vision main
md 267 lines 8.17 KB
Rendered Raw
1 <p align="center">
2 <img src="https://raw.githubusercontent.com/ondeinference/onde/main/assets/onde-inference-logo.svg" alt="Onde Inference" width="96">
3 </p>
4
5 <h1 align="center">Onde Inference</h1>
6
7 <p align="center">
8 <strong>Run LLMs on-device from Flutter and Dart. Metal on iOS and macOS, CPU everywhere else.</strong>
9 </p>
10
11 <p align="center">
12 <a href="https://pub.dev/packages/onde_inference"><img src="https://img.shields.io/pub/v/onde_inference?style=flat-square&color=235843&labelColor=17211D&label=pub.dev" alt="pub.dev"></a>
13 <a href="https://ondeinference.com"><img src="https://img.shields.io/badge/ondeinference.com-235843?style=flat-square&labelColor=17211D" alt="Website"></a>
14 <a href="https://github.com/ondeinference/onde/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-235843?style=flat-square&labelColor=17211D" alt="License"></a>
15 </p>
16
17 <p align="center">
18 <a href="https://github.com/ondeinference/onde">Rust SDK</a> · <a href="https://swiftpackageindex.com/ondeinference/onde-swift">Swift SDK</a> · <a href="https://central.sonatype.com/artifact/com.ondeinference/onde-inference">Kotlin Multiplatform SDK</a> · <a href="https://www.npmjs.com/package/@ondeinference/react-native">React Native SDK</a> · <a href="https://ondeinference.com">Website</a>
19 </p>
20
21 ---
22
23 Run Qwen 2.5 models directly inside your Flutter app. The model downloads from Hugging Face on first launch, then everything runs locally. No server, no API key, and no user data leaving the device. On an iPhone 15 Pro, Metal gets you around 15 tok/s. Android and desktop run on CPU, so they are slower, but they work well for local inference.
24
25 You get multi-turn chat, streaming, one-shot generation, and configurable sampling in a single package.
26
27 ## Platform support
28
29 | Platform | Backend | Default model | Notes |
30 |----------|---------|---------------|-------|
31 | iOS 13+ | Metal | Qwen 2.5 1.5B (~941 MB) | Simulator uses `aarch64-apple-ios-sim` |
32 | macOS 10.15+ | Metal | Qwen 2.5 3B (~1.93 GB) | Apple silicon and Intel |
33 | Android API 21+ | CPU | Qwen 2.5 1.5B (~941 MB) | arm64-v8a, armeabi-v7a, x86_64, x86 |
34 | Linux x86_64 | CPU | Qwen 2.5 3B (~1.93 GB) | CUDA possible, see docs |
35 | Windows x86_64 | CPU | Qwen 2.5 3B (~1.93 GB) | CUDA possible, see docs |
36
37 Web is not supported. On-device inference needs native system access that browsers don't expose.
38
39 ---
40
41 ## Quick start
42
43 ```yaml
44 dependencies:
45 onde_inference: ^0.1.0
46 ```
47
48 The inference engine is written in Rust and wired into Dart through [flutter_rust_bridge](https://pub.dev/packages/flutter_rust_bridge). You will need a working [Rust toolchain](https://rustup.rs). The first build is usually slow, around 5 to 10 minutes, because it has to compile the full dependency tree. After that, incremental builds are much faster.
49
50 ### Initialize
51
52 Call this once at startup, before anything else:
53
54 ```dart
55 import 'package:onde_inference/onde_inference.dart';
56
57 void main() async {
58 WidgetsFlutterBinding.ensureInitialized();
59 await OndeInference.init();
60 runApp(const MyApp());
61 }
62 ```
63
64 ### Load a model
65
66 ```dart
67 final engine = await OndeChatEngine.create();
68
69 // Picks the default model for the device:
70 // iOS / Android → Qwen 2.5 1.5B (~941 MB)
71 // macOS / Linux / Windows → Qwen 2.5 3B (~1.93 GB)
72 final elapsed = await engine.loadDefaultModel(
73 systemPrompt: 'You are a helpful assistant.',
74 );
75 print('Model loaded in ${elapsed.toStringAsFixed(1)} s');
76 ```
77
78 ### Chat
79
80 ```dart
81 final result = await engine.sendMessage('What is Rust's ownership model?');
82 print(result.text);
83 print('Generated in ${result.durationDisplay}');
84 ```
85
86 ### Stream
87
88 ```dart
89 final buffer = StringBuffer();
90
91 await for (final chunk in engine.streamMessage('Tell me a short story.')) {
92 buffer.write(chunk.delta);
93 setState(() => _displayText = buffer.toString());
94 if (chunk.done) break;
95 }
96 ```
97
98 ### Engine status
99
100 ```dart
101 final info = await engine.info();
102
103 print(info.status); // EngineStatus.ready
104 print(info.modelName); // "Qwen 2.5 3B"
105 print(info.approxMemory); // "~1.93 GB"
106 print(info.historyLength); // number of turns so far
107 ```
108
109 ### History
110
111 ```dart
112 final history = await engine.history();
113 for (final msg in history) {
114 print('${msg.role}: ${msg.content}');
115 }
116
117 // Clear history but keep the model loaded.
118 final removed = await engine.clearHistory();
119 print('Cleared $removed messages.');
120
121 // Seed from a saved session — no inference runs.
122 await engine.pushHistory(ChatMessage.user('Hello from last session!'));
123 await engine.pushHistory(ChatMessage.assistant('Hi! How can I help today?'));
124 ```
125
126 ### One-shot generation
127
128 This runs inference without changing conversation history. It is useful for things like prompt enhancement, classification, or formatting.
129
130 ```dart
131 final result = await engine.generate(
132 [
133 ChatMessage.system('You are a JSON formatter. Output only valid JSON.'),
134 ChatMessage.user('Name: Alice, Age: 30, City: Stockholm'),
135 ],
136 sampling: SamplingConfig.deterministic(),
137 );
138 print(result.text);
139 ```
140
141 ### Unload
142
143 ```dart
144 await engine.unloadModel();
145 ```
146
147 ---
148
149 ## Model selection
150
151 ```dart
152 // Platform-aware default (recommended).
153 final config = OndeInference.defaultModelConfig();
154
155 // Force a specific model.
156 final small = OndeInference.qwen251_5bConfig(); // ~941 MB
157 final medium = OndeInference.qwen253bConfig(); // ~1.93 GB
158 final coder = OndeInference.qwen25Coder3bConfig(); // ~1.93 GB, code-tuned
159
160 await engine.loadGgufModel(
161 medium,
162 systemPrompt: 'You are an expert software engineer.',
163 );
164 ```
165
166 | Model | Size | Good for |
167 |-------|------|----------|
168 | Qwen 2.5 1.5B Instruct Q4_K_M | ~941 MB | iOS, tvOS, Android |
169 | Qwen 2.5 3B Instruct Q4_K_M | ~1.93 GB | macOS, Linux, Windows |
170 | Qwen 2.5 Coder 1.5B Instruct Q4_K_M | ~941 MB | Code on mobile |
171 | Qwen 2.5 Coder 3B Instruct Q4_K_M | ~1.93 GB | Code on desktop |
172
173 ---
174
175 ## Sampling
176
177 All fields are optional. `null` means "use the engine default".
178
179 ```dart
180 final sampling = SamplingConfig(
181 temperature: 0.7,
182 topP: 0.95,
183 topK: 40,
184 maxTokens: 256,
185 );
186
187 await engine.setSampling(sampling);
188 ```
189
190 Presets:
191
192 ```dart
193 SamplingConfig.defaultConfig() // temp=0.7, max 512 tokens
194 SamplingConfig.deterministic() // greedy, temp=0.0
195 SamplingConfig.mobile() // temp=0.7, max 128 tokens
196 ```
197
198 ---
199
200 ## Error handling
201
202 All engine methods throw `OndeException` on failure:
203
204 ```dart
205 try {
206 await engine.loadDefaultModel();
207 } on OndeException catch (e) {
208 debugPrint('Inference error: ${e.message}');
209 }
210 ```
211
212 Common causes include calling `sendMessage` before loading a model, having no internet connection on first run while the model still needs to download, or running out of memory. The 3B model needs about 2 GB free, so on tighter devices you will usually want the 1.5B model instead.
213
214 ---
215
216 ## Sandboxed app setup (iOS / macOS)
217
218 On iOS and sandboxed macOS, the default Hugging Face cache path sits outside the app container. Call `setupCacheDir()` once at startup so Onde uses a location your app can actually access:
219
220 ```dart
221 import 'package:onde_inference/onde_inference.dart';
222 import 'package:path_provider/path_provider.dart';
223
224 void main() async {
225 WidgetsFlutterBinding.ensureInitialized();
226 await OndeInference.init();
227
228 String? fallback;
229 if (Platform.isIOS || Platform.isAndroid) {
230 final dir = await getApplicationSupportDirectory();
231 fallback = dir.path;
232 }
233 await OndeInference.setupCacheDir(fallbackDir: fallback);
234
235 runApp(const MyApp());
236 }
237 ```
238
239 Onde first tries the App Group shared container (`group.com.ondeinference.apps`) so Onde-powered apps can share downloaded models. If that App Group is not configured, it falls back to the app's private directory.
240
241 ---
242
243 ## Contributing
244
245 The source lives at [github.com/ondeinference/onde](https://github.com/ondeinference/onde):
246
247 - Rust core: `src/`
248 - Dart bridge crate: `sdk/dart/rust/`
249 - Dart library: `sdk/dart/lib/`
250 - Example app: `sdk/dart/example/`
251
252 Open an issue before sending large PRs.
253
254 ## License
255
256 Onde is dual-licensed under **MIT** and **Apache 2.0**. You can use either one.
257
258 - [MIT License](https://github.com/ondeinference/onde/blob/main/LICENSE-MIT)
259 - [Apache License 2.0](https://github.com/ondeinference/onde/blob/main/LICENSE-APACHE)
260
261 © 2026 [Splitfire AB](https://splitfire.se)
262
263 ---
264
265 ## Copyright
266
267 © 2026 [Onde Inference](https://ondeinference.com) (Splitfire AB).