|
1
|
<p align="center"> |
|
2
|
<img src="https://raw.githubusercontent.com/ondeinference/onde/main/assets/onde-inference-logo.svg" alt="Onde Inference" width="96"> |
|
3
|
</p> |
|
4
|
|
|
5
|
<h1 align="center">onde_inference — Example App</h1> |
|
6
|
|
|
7
|
<p align="center"> |
|
8
|
<strong>A Flutter chat app that runs LLM inference fully on-device.</strong><br> |
|
9
|
No server. No API key. No data leaving the device. |
|
10
|
</p> |
|
11
|
|
|
12
|
<p align="center"> |
|
13
|
<a href="https://pub.dev/packages/onde_inference"><img src="https://img.shields.io/pub/v/onde_inference?style=flat-square&color=235843&labelColor=17211D&label=pub.dev" alt="pub.dev"></a> |
|
14
|
<a href="https://ondeinference.com"><img src="https://img.shields.io/badge/ondeinference.com-235843?style=flat-square&labelColor=17211D" alt="Website"></a> |
|
15
|
<a href="https://apps.apple.com/se/developer/splitfire-ab/id1831430993"><img src="https://img.shields.io/badge/App%20Store-live-235843?style=flat-square&labelColor=17211D" alt="App Store"></a> |
|
16
|
</p> |
|
17
|
|
|
18
|
--- |
|
19
|
|
|
20
|
## What this example shows |
|
21
|
|
|
22
|
This example is a full Material 3 chat app built with the `onde_inference` Flutter SDK. It puts the main SDK features in one place, inside a single file, [`lib/main.dart`](lib/main.dart): |
|
23
|
|
|
24
|
| Feature | Where | |
|
25
|
|---|---| |
|
26
|
| `OndeInference.init()` + sandbox cache setup | `main()` | |
|
27
|
| Synchronous `OndeChatEngine()` factory (no `await`, no nulls) | `_ChatScreenState` | |
|
28
|
| Platform-aware default model loading (`loadDefaultModel`) | `_loadModel()` | |
|
29
|
| Multi-turn streaming chat via `streamMessage()` | `_sendMessage()` | |
|
30
|
| Live `EngineInfo` status bar (model name, memory, history length) | `_EngineStatusBar` | |
|
31
|
| `OndeError` sealed-class error handling | `_loadModel()`, `_sendMessage()` | |
|
32
|
| Sampling preset switcher (creative / precise / fast) | `_SamplingPreset` | |
|
33
|
| Unload and reload model at runtime | `_unloadModel()`, `_loadModel()` | |
|
34
|
| Blinking cursor animation during streaming | `_BlinkingCursor` | |
|
35
|
|
|
36
|
--- |
|
37
|
|
|
38
|
## Running the example |
|
39
|
|
|
40
|
### Prerequisites |
|
41
|
|
|
42
|
- [Flutter SDK](https://docs.flutter.dev/get-started/install) ≥ 3.10 |
|
43
|
- [Rust toolchain](https://rustup.rs) (stable), required to compile the native inference engine |
|
44
|
- A physical device or simulator/emulator for your target platform |
|
45
|
|
|
46
|
> **The first build usually takes 5 to 10 minutes** because it has to compile the full Rust dependency tree (`mistral.rs` + `candle`). After that, builds are much faster. |
|
47
|
|
|
48
|
### Steps |
|
49
|
|
|
50
|
```sh |
|
51
|
# From the repo root |
|
52
|
cd onde/sdk/dart/example |
|
53
|
|
|
54
|
# Install Dart dependencies |
|
55
|
flutter pub get |
|
56
|
|
|
57
|
# Run on your connected device or simulator |
|
58
|
flutter run |
|
59
|
``` |
|
60
|
|
|
61
|
On **iOS / macOS**, the Rust engine is built through the CocoaPods podspec and linked automatically by Xcode. On **Android**, a Gradle task cross-compiles it with `cargo-ndk` into `jniLibs`, so you need `cargo install cargo-ndk` plus the Android Rust targets. On **Linux / Windows**, it is linked as a shared library. |
|
62
|
|
|
63
|
Building all three Android ABIs takes a long time. Pass the ABI you actually |
|
64
|
need while iterating: |
|
65
|
|
|
66
|
```bash |
|
67
|
flutter run -d <device> --android-project-arg=onde.androidAbis=arm64-v8a |
|
68
|
``` |
|
69
|
|
|
70
|
### Platform notes |
|
71
|
|
|
72
|
| Platform | GPU backend | Default model loaded | |
|
73
|
|---|---|---| |
|
74
|
| iOS (device) | Metal | Qwen 2.5 1.5B (~941 MB) | |
|
75
|
| macOS | Metal | Qwen 2.5 3B (~1.93 GB) | |
|
76
|
| Android (arm64) | CPU | Qwen 2.5 1.5B (~941 MB) | |
|
77
|
| Linux / Windows | CPU | Qwen 2.5 3B (~1.93 GB) | |
|
78
|
|
|
79
|
> The model is downloaded from Hugging Face Hub on first launch and cached locally, so you only need an internet connection once. |
|
80
|
|
|
81
|
--- |
|
82
|
|
|
83
|
## SDK quick reference |
|
84
|
|
|
85
|
```dart |
|
86
|
import 'package:onde_inference/onde_inference.dart'; |
|
87
|
|
|
88
|
// 1. Initialise once at startup |
|
89
|
await OndeInference.init(); |
|
90
|
|
|
91
|
// 2. Create the engine and load the platform-appropriate model |
|
92
|
final engine = OndeChatEngine(); |
|
93
|
await engine.loadDefaultModel(systemPrompt: 'You are a helpful assistant.'); |
|
94
|
|
|
95
|
// 3. Stream a reply token-by-token |
|
96
|
await for (final chunk in engine.streamMessage('Hello!')) { |
|
97
|
stdout.write(chunk.delta); |
|
98
|
if (chunk.done) break; |
|
99
|
} |
|
100
|
|
|
101
|
// 4. Release memory when done |
|
102
|
await engine.unloadModel(); |
|
103
|
``` |
|
104
|
|
|
105
|
--- |
|
106
|
|
|
107
|
## Project structure |
|
108
|
|
|
109
|
``` |
|
110
|
example/ |
|
111
|
├── lib/ |
|
112
|
│ └── main.dart # Complete chat UI + SDK integration |
|
113
|
├── android/ # Android Gradle build |
|
114
|
├── ios/ # iOS Xcode project + CocoaPods |
|
115
|
├── macos/ # macOS Xcode project + CocoaPods |
|
116
|
├── linux/ # Linux CMake build |
|
117
|
└── windows/ # Windows CMake build |
|
118
|
``` |
|
119
|
|
|
120
|
--- |
|
121
|
|
|
122
|
## Learn more |
|
123
|
|
|
124
|
- **[onde_inference on pub.dev](https://pub.dev/packages/onde_inference)**, for the full API reference |
|
125
|
- **[Onde Inference docs](https://ondeinference.com)**, for guides, the model catalog, and platform setup |
|
126
|
- **[onde on GitHub](https://github.com/ondeinference/onde)**, for the Rust engine source, Swift SDK, and issue tracker |
|
127
|
|
|
128
|
--- |
|
129
|
|
|
130
|
## Copyright |
|
131
|
|
|
132
|
© 2026 [Splitfire AB](https://5mb.app) ([Onde Inference](https://ondeinference.com)). Licensed under the MIT License. |