|
1
|
<p align="center"> |
|
2
|
<img src="https://raw.githubusercontent.com/ondeinference/onde/main/assets/onde-inference-logo.svg" alt="Onde Inference" width="96"> |
|
3
|
</p> |
|
4
|
|
|
5
|
<h1 align="center">Onde Inference Kotlin SDK</h1> |
|
6
|
|
|
7
|
<p align="center"> |
|
8
|
<strong>Run LLMs on-device from Kotlin with <a href="https://ondeinference.com/">Onde Inference</a>. No cloud, no API key, and no user data leaving the device.</strong> |
|
9
|
</p> |
|
10
|
|
|
11
|
<p align="center"> |
|
12
|
<a href="https://central.sonatype.com/artifact/com.ondeinference/onde-inference"><img src="https://img.shields.io/maven-central/v/com.ondeinference/onde-inference?style=flat-square&color=235843&labelColor=17211D&label=maven" alt="Maven Central"></a> |
|
13
|
<a href="https://ondeinference.com"><img src="https://img.shields.io/badge/ondeinference.com-235843?style=flat-square&labelColor=17211D" alt="Website"></a> |
|
14
|
<a href="https://github.com/ondeinference/onde/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-235843?style=flat-square&labelColor=17211D" alt="License"></a> |
|
15
|
</p> |
|
16
|
|
|
17
|
<p align="center"> |
|
18
|
<a href="https://github.com/ondeinference/onde">Rust SDK</a> · <a href="https://github.com/ondeinference/onde-swift">Swift SDK</a> · <a href="https://pub.dev/packages/onde_inference">Flutter SDK</a> · <a href="https://www.npmjs.com/package/@ondeinference/react-native">React Native SDK</a> · <a href="https://ondeinference.com">Website</a> |
|
19
|
</p> |
|
20
|
|
|
21
|
--- |
|
22
|
|
|
23
|
## What is Onde? |
|
24
|
|
|
25
|
Onde is an on-device LLM inference SDK for Kotlin apps. It wraps the shared Rust core built on top of [mistral.rs](https://github.com/EricLBuehler/mistral.rs) in a Kotlin-friendly API with model downloads from Hugging Face, local cache management, and native inference under the hood. |
|
26
|
|
|
27
|
- **Runs locally**: no cloud, no API key, no user data leaving the device |
|
28
|
- **Kotlin-friendly**: `suspend` functions and `Flow<StreamChunk>` for streaming |
|
29
|
- **Shared Rust core**: the same engine powers the Rust, Swift, Flutter, and React Native SDKs |
|
30
|
- **Kotlin Multiplatform**: today this package targets Android and JVM |
|
31
|
- **Published on Maven Central**: add one Gradle dependency and start integrating |
|
32
|
|
|
33
|
If you want to experiment with models before wiring them into an app, use [Onde CLI](https://github.com/ondeinference/onde-cli). |
|
34
|
|
|
35
|
--- |
|
36
|
|
|
37
|
## Installation |
|
38
|
|
|
39
|
Add the SDK to your app's `build.gradle.kts`: |
|
40
|
|
|
41
|
```kotlin |
|
42
|
dependencies { |
|
43
|
implementation("com.ondeinference:onde-inference:1.0.0") |
|
44
|
} |
|
45
|
``` |
|
46
|
|
|
47
|
Add `INTERNET` permission to `AndroidManifest.xml` for the initial model download: |
|
48
|
|
|
49
|
```xml |
|
50
|
<uses-permission android:name="android.permission.INTERNET" /> |
|
51
|
``` |
|
52
|
|
|
53
|
### Minimum requirements |
|
54
|
|
|
55
|
- **Android**: API 26+ (Android 8.0+) |
|
56
|
- **Storage**: about 1.1 GB free for the default Android model |
|
57
|
- **JVM**: a supported desktop target with the bundled native Onde library |
|
58
|
|
|
59
|
--- |
|
60
|
|
|
61
|
## Quick start on Android |
|
62
|
|
|
63
|
```kotlin |
|
64
|
import android.os.Bundle |
|
65
|
import androidx.activity.ComponentActivity |
|
66
|
import androidx.lifecycle.lifecycleScope |
|
67
|
import com.ondeinference.onde.OndeInference |
|
68
|
import kotlinx.coroutines.launch |
|
69
|
|
|
70
|
class MainActivity : ComponentActivity() { |
|
71
|
private val onde by lazy { OndeInference(applicationContext) } |
|
72
|
|
|
73
|
override fun onCreate(savedInstanceState: Bundle?) { |
|
74
|
super.onCreate(savedInstanceState) |
|
75
|
|
|
76
|
lifecycleScope.launch { |
|
77
|
val elapsed = onde.loadDefaultModel( |
|
78
|
systemPrompt = "You are a helpful, concise assistant." |
|
79
|
) |
|
80
|
println("Model loaded in ${elapsed}s") |
|
81
|
|
|
82
|
val result = onde.chat("What is the capital of Sweden?") |
|
83
|
println(result.text) |
|
84
|
} |
|
85
|
} |
|
86
|
|
|
87
|
override fun onDestroy() { |
|
88
|
super.onDestroy() |
|
89
|
onde.close() |
|
90
|
} |
|
91
|
} |
|
92
|
``` |
|
93
|
|
|
94
|
On Android, the default model is **Qwen 2.5 1.5B Instruct (GGUF Q4_K_M, ~941 MB)**. |
|
95
|
|
|
96
|
--- |
|
97
|
|
|
98
|
## Quick start on JVM |
|
99
|
|
|
100
|
```kotlin |
|
101
|
import com.ondeinference.onde.OndeInference |
|
102
|
import kotlinx.coroutines.runBlocking |
|
103
|
|
|
104
|
fun main() = runBlocking { |
|
105
|
val onde = OndeInference() |
|
106
|
try { |
|
107
|
onde.loadDefaultModel(systemPrompt = "You are a helpful assistant.") |
|
108
|
val reply = onde.chat("Give me three Kotlin tips.") |
|
109
|
println(reply.text) |
|
110
|
} finally { |
|
111
|
onde.close() |
|
112
|
} |
|
113
|
} |
|
114
|
``` |
|
115
|
|
|
116
|
On desktop JVM targets, the default model is **Qwen 2.5 3B Instruct (GGUF Q4_K_M, ~1.93 GB)**. |
|
117
|
|
|
118
|
--- |
|
119
|
|
|
120
|
## Streaming |
|
121
|
|
|
122
|
If you want tokens as they arrive, collect a `Flow<StreamChunk>`: |
|
123
|
|
|
124
|
```kotlin |
|
125
|
lifecycleScope.launch { |
|
126
|
onde.stream("Write a haiku about the ocean.").collect { chunk -> |
|
127
|
textView.append(chunk.delta) |
|
128
|
if (chunk.done) println("\n[done]") |
|
129
|
} |
|
130
|
} |
|
131
|
``` |
|
132
|
|
|
133
|
--- |
|
134
|
|
|
135
|
## Multi-turn conversation |
|
136
|
|
|
137
|
Onde keeps conversation history for you: |
|
138
|
|
|
139
|
```kotlin |
|
140
|
lifecycleScope.launch { |
|
141
|
onde.loadDefaultModel(systemPrompt = "You are a Rust tutor.") |
|
142
|
|
|
143
|
val r1 = onde.chat("What is ownership?") |
|
144
|
println(r1.text) |
|
145
|
|
|
146
|
val r2 = onde.chat("Can you give me a code example?") |
|
147
|
println(r2.text) |
|
148
|
|
|
149
|
onde.clearHistory() |
|
150
|
} |
|
151
|
``` |
|
152
|
|
|
153
|
--- |
|
154
|
|
|
155
|
## Models and sampling |
|
156
|
|
|
157
|
The SDK exposes convenience helpers for supported models and sampling presets: |
|
158
|
|
|
159
|
```kotlin |
|
160
|
import com.ondeinference.onde.OndeModels |
|
161
|
import com.ondeinference.onde.OndeSampling |
|
162
|
|
|
163
|
lifecycleScope.launch { |
|
164
|
onde.loadModel( |
|
165
|
config = OndeModels.qwen25_1_5b(), |
|
166
|
systemPrompt = "You are a coding assistant.", |
|
167
|
sampling = OndeSampling.deterministic(), |
|
168
|
) |
|
169
|
} |
|
170
|
``` |
|
171
|
|
|
172
|
Available helpers include: |
|
173
|
|
|
174
|
- `OndeModels.default()` |
|
175
|
- `OndeModels.qwen25_1_5b()` |
|
176
|
- `OndeModels.qwen25_3b()` |
|
177
|
- `OndeSampling.default()` |
|
178
|
- `OndeSampling.deterministic()` |
|
179
|
- `OndeSampling.mobile()` |
|
180
|
|
|
181
|
--- |
|
182
|
|
|
183
|
## One-shot generation |
|
184
|
|
|
185
|
Use `generate()` when you want a response without mutating chat history: |
|
186
|
|
|
187
|
```kotlin |
|
188
|
import com.ondeinference.onde.OndeMessage |
|
189
|
import com.ondeinference.onde.OndeSampling |
|
190
|
|
|
191
|
lifecycleScope.launch { |
|
192
|
val result = onde.generate( |
|
193
|
messages = listOf( |
|
194
|
OndeMessage.system("You are a summariser."), |
|
195
|
OndeMessage.user("Summarise this article in five bullet points."), |
|
196
|
), |
|
197
|
sampling = OndeSampling.deterministic(), |
|
198
|
) |
|
199
|
println(result.text) |
|
200
|
} |
|
201
|
``` |
|
202
|
|
|
203
|
--- |
|
204
|
|
|
205
|
## Engine status and cache location |
|
206
|
|
|
207
|
```kotlin |
|
208
|
lifecycleScope.launch { |
|
209
|
val info = onde.info() |
|
210
|
println("Status: ${info.status}") |
|
211
|
println("Model: ${info.modelName}") |
|
212
|
println("History: ${info.historyLength} turns") |
|
213
|
} |
|
214
|
|
|
215
|
println(onde.modelCacheDir.absolutePath) |
|
216
|
``` |
|
217
|
|
|
218
|
Onde stores model files inside the configured data directory: |
|
219
|
|
|
220
|
```text |
|
221
|
<dataDir>/ |
|
222
|
├── models/ |
|
223
|
│ └── hub/ |
|
224
|
│ └── models--bartowski--Qwen2.5-1.5B-Instruct-GGUF/ |
|
225
|
│ └── ... |
|
226
|
└── tmp/ |
|
227
|
``` |
|
228
|
|
|
229
|
On Android, the default `dataDir` is `context.filesDir`. |
|
230
|
|
|
231
|
--- |
|
232
|
|
|
233
|
## Building from source |
|
234
|
|
|
235
|
The Kotlin SDK lives in `sdk/kotlin/` and is built from the shared Rust crate in the repository root. |
|
236
|
|
|
237
|
### Prerequisites |
|
238
|
|
|
239
|
```bash |
|
240
|
# Rust Android targets |
|
241
|
rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android i686-linux-android |
|
242
|
|
|
243
|
# cargo-ndk for Android cross-compilation |
|
244
|
cargo install cargo-ndk |
|
245
|
|
|
246
|
# Android NDK |
|
247
|
export ANDROID_NDK_HOME=$HOME/Library/Android/sdk/ndk/26.1.10909125 |
|
248
|
``` |
|
249
|
|
|
250
|
### Build Android native libraries |
|
251
|
|
|
252
|
```bash |
|
253
|
./sdk/kotlin/scripts/build-android.sh |
|
254
|
``` |
|
255
|
|
|
256
|
This produces `.so` files in `sdk/kotlin/lib/src/androidMain/jniLibs/`. |
|
257
|
|
|
258
|
### Build JVM native library |
|
259
|
|
|
260
|
```bash |
|
261
|
./sdk/kotlin/scripts/build-jvm.sh |
|
262
|
``` |
|
263
|
|
|
264
|
This bundles the desktop native library into `sdk/kotlin/lib/src/jvmMain/resources/native/`. |
|
265
|
|
|
266
|
### Generate Kotlin UniFFI bindings |
|
267
|
|
|
268
|
```bash |
|
269
|
./sdk/kotlin/scripts/generate-bindings.sh |
|
270
|
``` |
|
271
|
|
|
272
|
This generates the Kotlin bindings in `sdk/kotlin/lib/src/generated/kotlin/`. |
|
273
|
|
|
274
|
> The generated bindings and compiled native libraries are intentionally gitignored and should be regenerated from source. |
|
275
|
|
|
276
|
### Build the Gradle modules |
|
277
|
|
|
278
|
```bash |
|
279
|
cd sdk/kotlin |
|
280
|
./gradlew :lib:assembleRelease |
|
281
|
./gradlew :example:assembleDebug |
|
282
|
``` |
|
283
|
|
|
284
|
--- |
|
285
|
|
|
286
|
## Publishing to Maven Central |
|
287
|
|
|
288
|
Publishing is handled by CI. The published coordinates are controlled by `sdk/kotlin/gradle.properties`: |
|
289
|
|
|
290
|
- `GROUP=com.ondeinference` |
|
291
|
- `POM_ARTIFACT_ID=onde-inference` |
|
292
|
- `VERSION_NAME=1.0.0` |
|
293
|
|
|
294
|
Tag a release after bumping the Rust and Kotlin versions together. |
|
295
|
|
|
296
|
--- |
|
297
|
|
|
298
|
## Project layout |
|
299
|
|
|
300
|
```text |
|
301
|
sdk/kotlin/ |
|
302
|
├── build.gradle.kts |
|
303
|
├── settings.gradle.kts |
|
304
|
├── gradle.properties |
|
305
|
├── README.md |
|
306
|
├── example/ # Android demo app |
|
307
|
├── lib/ # Kotlin Multiplatform SDK module |
|
308
|
│ └── src/ |
|
309
|
│ ├── androidMain/ |
|
310
|
│ ├── jvmMain/ |
|
311
|
│ ├── shared/ |
|
312
|
│ └── generated/ |
|
313
|
└── scripts/ |
|
314
|
├── build-android.sh |
|
315
|
├── build-jvm.sh |
|
316
|
└── generate-bindings.sh |
|
317
|
``` |
|
318
|
|
|
319
|
--- |
|
320
|
|
|
321
|
## Troubleshooting |
|
322
|
|
|
323
|
**Download never finishes** — check your internet connection. Hugging Face Hub can rate-limit anonymous downloads. |
|
324
|
|
|
325
|
**App crashes immediately on Android** — make sure the device is running API 26 or newer. The Android platform setup uses `Os.setenv`. |
|
326
|
|
|
327
|
**Inference is very slow** — Android emulators are slow for CPU inference. A physical ARM device is much faster. |
|
328
|
|
|
329
|
**Out of memory while loading** — the default Android model needs roughly 1–1.5 GB of free RAM. |
|
330
|
|
|
331
|
**Native library not found on desktop** — rebuild the JVM native library with `./sdk/kotlin/scripts/build-jvm.sh` so the expected resource bundle is present. |
|
332
|
|
|
333
|
--- |
|
334
|
|
|
335
|
## License |
|
336
|
|
|
337
|
MIT OR Apache-2.0. See [LICENSE](../../LICENSE). |
|
338
|
|
|
339
|
## Copyright |
|
340
|
|
|
341
|
© 2026 [Splitfire AB](https://5mb.app) ([Onde Inference](https://ondeinference.com)). |