|
1
|
<p align="center"> |
|
2
|
<img src="https://raw.githubusercontent.com/ondeinference/onde/main/assets/onde-inference-logo.svg" alt="Onde Inference" width="96"> |
|
3
|
</p> |
|
4
|
|
|
5
|
<h1 align="center">Onde Inference</h1> |
|
6
|
|
|
7
|
<p align="center"> |
|
8
|
<strong>Run LLMs on-device from React Native with <a href="https://ondeinference.com/">Onde Inference</a>. Metal on iOS, CPU on Android, and no user data leaving the device.</strong> |
|
9
|
</p> |
|
10
|
|
|
11
|
<p align="center"> |
|
12
|
<a href="https://www.npmjs.com/package/@ondeinference/react-native"><img src="https://img.shields.io/npm/v/@ondeinference/react-native?style=flat-square&color=235843&labelColor=17211D&label=npm" alt="npm"></a> |
|
13
|
<a href="https://ondeinference.com"><img src="https://img.shields.io/badge/ondeinference.com-235843?style=flat-square&labelColor=17211D" alt="Website"></a> |
|
14
|
<a href="https://github.com/ondeinference/onde/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-235843?style=flat-square&labelColor=17211D" alt="License"></a> |
|
15
|
</p> |
|
16
|
|
|
17
|
<p align="center"> |
|
18
|
<a href="https://github.com/ondeinference/onde">Rust SDK</a> · <a href="https://swiftpackageindex.com/ondeinference/onde-swift">Swift SDK</a> · <a href="https://central.sonatype.com/artifact/com.ondeinference/onde-inference">Kotlin Multiplatform SDK</a> · <a href="https://pub.dev/packages/onde_inference">Flutter SDK</a> · <a href="https://ondeinference.com">Website</a> |
|
19
|
</p> |
|
20
|
|
|
21
|
--- |
|
22
|
|
|
23
|
Run Qwen 2.5 models directly on the device. No server, no API key, and no user data leaving the phone. For an [efficient on-device inference engine for React Native](https://ondeinference.com/sdk/react-native), the SDK page is the quickest place to check install details and platform notes. If you want to verify model downloads or GGUF output before you ship the app build, use [Onde CLI](https://github.com/ondeinference/onde-cli). |
|
24
|
|
|
25
|
The model downloads from Hugging Face the first time you load it, then runs locally after that. The 1.5B model is about 941 MB. On iPhone, Metal makes it feel surprisingly fast. On Android, it runs on CPU, so it is slower, but it still works well enough for local chat. |
|
26
|
|
|
27
|
## Installation |
|
28
|
|
|
29
|
```bash |
|
30
|
npx expo install @ondeinference/react-native |
|
31
|
``` |
|
32
|
|
|
33
|
## Quick start |
|
34
|
|
|
35
|
```typescript |
|
36
|
import { OndeChatEngine, userMessage } from "@ondeinference/react-native"; |
|
37
|
|
|
38
|
// Picks the default model for the device: |
|
39
|
// iOS → Qwen 2.5 1.5B (~941 MB, Metal) |
|
40
|
// Android → Qwen 2.5 1.5B (~941 MB, CPU) |
|
41
|
const seconds = await OndeChatEngine.loadDefaultModel( |
|
42
|
"You are a helpful assistant." |
|
43
|
); |
|
44
|
|
|
45
|
const reply = await OndeChatEngine.sendMessage("Hello!"); |
|
46
|
console.log(reply.text); |
|
47
|
|
|
48
|
// One-shot — doesn't touch conversation history |
|
49
|
const expanded = await OndeChatEngine.generate( |
|
50
|
[userMessage("Expand: a cat in space")], |
|
51
|
{ temperature: 0.0 } |
|
52
|
); |
|
53
|
|
|
54
|
await OndeChatEngine.unloadModel(); |
|
55
|
``` |
|
56
|
|
|
57
|
## Platforms |
|
58
|
|
|
59
|
| Platform | Backend | Default model | |
|
60
|
|----------|---------|---------------| |
|
61
|
| iOS | Metal | Qwen 2.5 1.5B (~941 MB) | |
|
62
|
| Android | CPU | Qwen 2.5 1.5B (~941 MB) | |
|
63
|
|
|
64
|
## API |
|
65
|
|
|
66
|
### OndeChatEngine |
|
67
|
|
|
68
|
| Method | Returns | What it does | |
|
69
|
|--------|---------|--------------| |
|
70
|
| `loadDefaultModel(systemPrompt?, sampling?)` | `Promise<number>` | Load the platform default. Returns load time in seconds. | |
|
71
|
| `loadModel(config, systemPrompt?, sampling?)` | `Promise<number>` | Load a specific GGUF model. | |
|
72
|
| `loadUqffModel(config, systemPrompt?, sampling?)` | `Promise<number>` | Load a specific UQFF model. | |
|
73
|
| `unloadModel()` | `Promise<string \| null>` | Drop the model, free memory. Returns the model name. | |
|
74
|
| `isLoaded()` | `boolean` | Is anything loaded right now? | |
|
75
|
| `info()` | `Promise<EngineInfo>` | Status, model name, memory, history length. | |
|
76
|
| `sendMessage(message)` | `Promise<InferenceResult>` | Chat turn. Appends to history automatically. | |
|
77
|
| `generate(messages, sampling?)` | `Promise<InferenceResult>` | One-shot. History stays untouched. | |
|
78
|
| `setSystemPrompt(prompt)` | `void` | Replace the system prompt. | |
|
79
|
| `clearSystemPrompt()` | `void` | Remove it. | |
|
80
|
| `setSampling(config)` | `void` | Swap sampling params. | |
|
81
|
| `history()` | `Promise<ChatMessage[]>` | Full conversation so far. | |
|
82
|
| `clearHistory()` | `number` | Wipe it. Returns how many messages were removed. | |
|
83
|
| `pushHistory(message)` | `void` | Inject a message without running inference. | |
|
84
|
|
|
85
|
### Helpers |
|
86
|
|
|
87
|
```typescript |
|
88
|
import { |
|
89
|
defaultModelConfig, // platform-aware (1.5B on mobile, 3B on desktop) |
|
90
|
uqffModelConfig, // base repo/local dir + UQFF shard(s) |
|
91
|
qwen251_5bConfig, // force 1.5B (~941 MB) |
|
92
|
qwen253bConfig, // force 3B (~1.93 GB) |
|
93
|
defaultSamplingConfig, // temp=0.7, top_p=0.95, max_tokens=512 |
|
94
|
deterministicSamplingConfig, // temp=0.0 |
|
95
|
mobileSamplingConfig, // temp=0.7, max_tokens=128 |
|
96
|
systemMessage, |
|
97
|
userMessage, |
|
98
|
assistantMessage, |
|
99
|
} from "@ondeinference/react-native"; |
|
100
|
``` |
|
101
|
|
|
102
|
UQFF example: |
|
103
|
|
|
104
|
```typescript |
|
105
|
const config = uqffModelConfig( |
|
106
|
"google/gemma-4-E4B-it", |
|
107
|
["q4k-0.uqff"], |
|
108
|
"Gemma 4 E4B (UQFF Q4K)", |
|
109
|
"~2.5 GB (UQFF Q4K)", |
|
110
|
); |
|
111
|
|
|
112
|
await OndeChatEngine.loadUqffModel(config); |
|
113
|
``` |
|
114
|
|
|
115
|
## Example app |
|
116
|
|
|
117
|
There is a working chat app in [`example/`](./example): |
|
118
|
|
|
119
|
```bash |
|
120
|
cd example |
|
121
|
npm install |
|
122
|
npx expo run:ios |
|
123
|
``` |
|
124
|
|
|
125
|
It is a single-file example, about 290 lines, and it covers loading, chat, status, history management, and error handling. |
|
126
|
|
|
127
|
## Building from source |
|
128
|
|
|
129
|
You need Rust and the right cross-compilation targets. |
|
130
|
|
|
131
|
```bash |
|
132
|
# iOS |
|
133
|
rustup target add aarch64-apple-ios aarch64-apple-ios-sim |
|
134
|
./scripts/build-rust.sh ios |
|
135
|
|
|
136
|
# Android (set ANDROID_NDK_HOME first) |
|
137
|
rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android i686-linux-android |
|
138
|
./scripts/build-rust.sh android |
|
139
|
``` |
|
140
|
|
|
141
|
The script builds the Rust FFI bridge in `rust/`, then copies the static library for iOS or the shared libraries for Android into the right places under `ios/` and `android/`. |
|
142
|
|
|
143
|
## How it fits together |
|
144
|
|
|
145
|
``` |
|
146
|
TypeScript → Expo Module (Swift / Kotlin) → Rust C FFI → onde crate → mistral.rs |
|
147
|
@_silgen_name (iOS) ↓ |
|
148
|
JNI external (Android) Metal / CPU |
|
149
|
``` |
|
150
|
|
|
151
|
The native module talks to Rust through `extern "C"` functions. Complex types cross the boundary as JSON strings, and the TypeScript layer handles camelCase ↔ snake_case conversion. A global `tokio::Runtime`, created once, runs the async inference work. |
|
152
|
|
|
153
|
## License |
|
154
|
|
|
155
|
Onde is dual-licensed under **MIT** and **Apache 2.0**. You can use either one. |
|
156
|
|
|
157
|
- [MIT License](https://github.com/ondeinference/onde/blob/main/LICENSE-MIT) |
|
158
|
- [Apache License 2.0](https://github.com/ondeinference/onde/blob/main/LICENSE-APACHE) |
|
159
|
|
|
160
|
--- |
|
161
|
|
|
162
|
## Copyright |
|
163
|
|
|
164
|
© 2026 [Splitfire AB](https://5mb.app) ([Onde Inference](https://ondeinference.com)). |