@ondeinference / onde
Code Commits Issues Pulls Cloud Agent CI/CD
copilot/fix-onde-npm-sdk-v1-1-2-release
copilot/assess-inference-engine-complexity copilot/fix-onde-npm-sdk-v1-1-2-release development feature/diffusion-vision main
md 331 lines 9.82 KB
Rendered Raw
1 <p align="center">
2 <img src="https://raw.githubusercontent.com/ondeinference/onde/main/assets/onde-inference-logo.svg" alt="Onde Inference" width="96">
3 </p>
4
5 <h1 align="center">Onde Inference Kotlin SDK</h1>
6
7 <p align="center">
8 <strong>Run LLMs on-device from Kotlin with <a href="https://ondeinference.com/">Onde Inference</a>. No cloud, no API key, and no user data leaving the device.</strong>
9 </p>
10
11 <p align="center">
12 <a href="https://central.sonatype.com/artifact/com.ondeinference/onde-inference"><img src="https://img.shields.io/maven-central/v/com.ondeinference/onde-inference?style=flat-square&color=235843&labelColor=17211D&label=maven" alt="Maven Central"></a>
13 <a href="https://ondeinference.com"><img src="https://img.shields.io/badge/ondeinference.com-235843?style=flat-square&labelColor=17211D" alt="Website"></a>
14 <a href="https://github.com/ondeinference/onde/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-235843?style=flat-square&labelColor=17211D" alt="License"></a>
15 </p>
16
17 <p align="center">
18 <a href="https://github.com/ondeinference/onde">Rust SDK</a> · <a href="https://github.com/ondeinference/onde-swift">Swift SDK</a> · <a href="https://pub.dev/packages/onde_inference">Flutter SDK</a> · <a href="https://www.npmjs.com/package/@ondeinference/react-native">React Native SDK</a> · <a href="https://ondeinference.com">Website</a>
19 </p>
20
21 ---
22
23 ## What is Onde?
24
25 Onde is an on-device LLM inference SDK for Kotlin apps. It wraps [mistral.rs](https://github.com/EricLBuehler/mistral.rs) in a Kotlin-friendly API, with model downloads from Hugging Face, local cache management, and native inference under the hood.
26
27 - **Runs locally**: no cloud, no API key, no user data leaving the device
28 - **Kotlin-friendly**: suspend functions and `Flow<StreamChunk>` for streaming
29 - **Rust underneath**: powered by [uniffi-rs](https://github.com/mozilla/uniffi-rs)
30 - **Published on Maven Central**: add one Gradle dependency and you're done
31
32 This package is built as a Kotlin Multiplatform library. Today that means Android and JVM support, with the same Rust core underneath both. For [on-device AI inference for Kotlin and Android](https://ondeinference.com/sdk), the SDK page is the quickest way to check package docs and platform scope. If you want to test model downloads or fine-tune outputs before moving into app code, use [Onde CLI](https://github.com/ondeinference/onde-cli).
33
34 ---
35
36 ## Installation
37
38 Add to your app's `build.gradle.kts`:
39
40 ```kotlin
41 dependencies {
42 implementation("com.ondeinference:onde-inference:0.1.3")
43 }
44 ```
45
46 Add `INTERNET` permission to `AndroidManifest.xml` (required for the initial model download):
47
48 ```xml
49 <uses-permission android:name="android.permission.INTERNET" />
50 ```
51
52 **Minimum requirements**
53 - Android 8.0 (API 26) or higher
54 - About 1.1 GB of free storage for the default model (Qwen 2.5 1.5B Q4_K_M)
55
56 ---
57
58 ## Quick start on Android
59
60 ```kotlin
61 import com.ondeinference.onde.OndeInference
62 import kotlinx.coroutines.launch
63
64 class MainActivity : AppCompatActivity() {
65
66 private val onde by lazy { OndeInference(applicationContext) }
67
68 override fun onCreate(savedInstanceState: Bundle?) {
69 super.onCreate(savedInstanceState)
70
71 lifecycleScope.launch {
72 // 1. Load the default model (Qwen 2.5 1.5B on Android, ~941 MB)
73 // It downloads from Hugging Face the first time you run it.
74 val elapsed = onde.loadDefaultModel(
75 systemPrompt = "You are a helpful, concise assistant."
76 )
77 println("Model loaded in ${elapsed}s")
78
79 // 2. Chat
80 val result = onde.chat("What is the capital of Sweden?")
81 println(result.text) // → "The capital of Sweden is Stockholm."
82
83 // 3. Clean up
84 onde.unload()
85 }
86 }
87
88 override fun onDestroy() {
89 super.onDestroy()
90 onde.close()
91 }
92 }
93 ```
94
95 ---
96
97 ## Streaming
98
99 If you want tokens as they arrive, use `Flow`:
100
101 ```kotlin
102 lifecycleScope.launch {
103 onde.stream("Write a haiku about the ocean.").collect { chunk ->
104 // Append each token delta to your UI
105 textView.append(chunk.delta)
106 if (chunk.done) println("\n[done]")
107 }
108 }
109 ```
110
111 ---
112
113 ## Multi-turn conversation
114
115 Onde keeps conversation history for you:
116
117 ```kotlin
118 lifecycleScope.launch {
119 onde.loadDefaultModel(systemPrompt = "You are a Rust tutor.")
120
121 val r1 = onde.chat("What is ownership?")
122 println(r1.text)
123
124 val r2 = onde.chat("Can you give me a code example?")
125 println(r2.text) // knows the context from r1
126
127 // Clear history without unloading the model
128 onde.clearHistory()
129 }
130 ```
131
132 ---
133
134 ## Load a specific model
135
136 ```kotlin
137 import com.ondeinference.onde.OndeModels
138
139 lifecycleScope.launch {
140 // Qwen 2.5 1.5B, the default on Android
141 val elapsed = onde.loadModel(
142 config = OndeModels.qwen25_1_5b(),
143 systemPrompt = "You are a coding assistant."
144 )
145 }
146 ```
147
148 ---
149
150 ## Sampling configuration
151
152 ```kotlin
153 import com.ondeinference.onde.OndeSampling
154
155 lifecycleScope.launch {
156 // Default chat settings: temperature 0.7, top_p 0.95, max 512 tokens
157 onde.loadDefaultModel(sampling = OndeSampling.default())
158
159 // Deterministic output, useful for factual answers and code
160 onde.loadDefaultModel(sampling = OndeSampling.deterministic())
161
162 // Shorter responses, usually better for lower-end devices
163 onde.loadDefaultModel(sampling = OndeSampling.mobile())
164 }
165 ```
166
167 ---
168
169 ## One-shot generation (without changing chat history)
170
171 ```kotlin
172 import com.ondeinference.onde.OndeMessage
173
174 lifecycleScope.launch {
175 val result = onde.generate(
176 messages = listOf(
177 OndeMessage.system("You are a summariser."),
178 OndeMessage.user("Summarise this text: $longArticle"),
179 ),
180 sampling = OndeSampling.deterministic()
181 )
182 println(result.text)
183 }
184 ```
185
186 ---
187
188 ## Engine status
189
190 ```kotlin
191 lifecycleScope.launch {
192 val info = onde.info()
193 println("Status: ${info.status}") // Ready / Loading / Unloaded / Error
194 println("Model: ${info.modelName}")
195 println("Memory: ${info.approxMemory}")
196 println("History: ${info.historyLength} turns")
197 }
198 ```
199
200 ---
201
202 ## File system layout
203
204 Onde stores model files in the app's internal `filesDir`. You do not need external
205 storage permissions. `INTERNET` is only needed for the initial download.
206
207 ```
208 <filesDir>/
209 ├── models/
210 │ └── hub/
211 │ └── models--bartowski--Qwen2.5-1.5B-Instruct-GGUF/
212 │ └── ...
213 └── tmp/
214 ```
215
216 You can inspect the cache location at runtime:
217
218 ```kotlin
219 println(onde.modelCacheDir.absolutePath)
220 ```
221
222 ---
223
224 ## How it fits together
225
226 ```
227 ┌───────────────────────────────────────┐
228 │ Kotlin app / Android activity │
229 │ import com.ondeinference.onde.* │
230 └──────────────┬────────────────────────┘
231 │ suspend / Flow
232 ▼
233 ┌───────────────────────────────────────┐
234 │ OndeInference.kt (this SDK) │
235 │ Android filesystem setup │
236 │ Dispatchers.IO dispatch │
237 └──────────────┬────────────────────────┘
238 │ UniFFI-generated Kotlin (onde.kt)
239 ▼
240 ┌───────────────────────────────────────┐
241 │ libonde.so (Rust, per ABI) │
242 │ mistral.rs + Candle (CPU) │
243 │ HuggingFace Hub download │
244 └───────────────────────────────────────┘
245 ```
246
247 On Android, the Rust engine is compiled for four ABIs and bundled as JNI libs:
248
249 | ABI | Architecture |
250 |-----|-------------|
251 | `arm64-v8a` | 64-bit ARM (modern Android phones) |
252 | `armeabi-v7a` | 32-bit ARM (older phones) |
253 | `x86_64` | Android emulators (Intel/AMD) |
254 | `x86` | 32-bit emulators |
255
256 ---
257
258 ## Building from source
259
260 ### Prerequisites
261
262 ```bash
263 # Rust Android targets
264 rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android i686-linux-android
265
266 # cargo-ndk — manages NDK toolchain for cross-compilation
267 cargo install cargo-ndk
268
269 # Android NDK (via Android Studio or SDK manager)
270 export ANDROID_NDK_HOME=$HOME/Library/Android/sdk/ndk/26.1.10909125
271 ```
272
273 ### Build the native libraries
274
275 ```bash
276 # From the repo root
277 ./sdk/kotlin/scripts/build-android.sh
278 ```
279
280 This produces `.so` files in `sdk/kotlin/lib/src/androidMain/jniLibs/`.
281
282 ### Generate Kotlin UniFFI bindings
283
284 ```bash
285 ./sdk/kotlin/scripts/generate-bindings.sh
286 ```
287
288 This generates `onde.kt` in `sdk/kotlin/lib/src/generated/kotlin/`. The generated file
289 is gitignored, so regenerate it whenever the Rust API changes.
290
291 ### Build the Android artifact
292
293 ```bash
294 cd sdk/kotlin
295 ./gradlew :lib:assembleRelease
296 # Output: lib/build/outputs/aar/lib-release.aar
297 ```
298
299 ---
300
301 ## Publishing to Maven Central
302
303 Publishing is handled by CI. It requires:
304
305 | Secret | Description |
306 |--------|-------------|
307 | `MAVEN_CENTRAL_USERNAME` | Sonatype Central Portal username |
308 | `MAVEN_CENTRAL_PASSWORD` | Sonatype Central Portal password |
309 | `SIGNING_KEY_ID` | PGP key ID (last 8 hex chars) |
310 | `SIGNING_KEY` | ASCII-armored PGP private key |
311 | `SIGNING_PASSWORD` | PGP key passphrase |
312
313 Tag a release to trigger the workflow:
314
315 ```bash
316 # Bump version in Cargo.toml and gradle.properties first
317 git tag 0.1.4 && git push origin 0.1.4
318 ```
319
320 The CI workflow builds the native libraries, generates bindings, packages the Android
321 artifact, signs it, and publishes to Maven Central automatically.
322
323 ---
324
325 ## License
326
327 MIT OR Apache-2.0. See [LICENSE](../../LICENSE).
328
329 ## Copyright
330
331 © 2026 [Onde Inference](https://ondeinference.com) (Splitfire AB).