|
1
|
# Onde Chat |
|
2
|
|
|
3
|
A simple Android chat app built with the [Onde Inference](https://ondeinference.com) SDK. It runs Qwen 2.5 1.5B entirely on your phone — no server, no API key, nothing leaves the device. |
|
4
|
|
|
5
|
--- |
|
6
|
|
|
7
|
## What it does |
|
8
|
|
|
9
|
- Downloads the model on first launch (~941 MB, cached after that) |
|
10
|
- Streams replies token by token as the model generates them |
|
11
|
- Keeps a full conversation history across turns |
|
12
|
- Works offline once the model is cached |
|
13
|
|
|
14
|
--- |
|
15
|
|
|
16
|
## Getting started |
|
17
|
|
|
18
|
Open `onde/sdk/kotlin/example/` as the project root in Android Studio (not the repo root — the `settings.gradle.kts` lives here). Gradle will pull everything from Maven Central on first sync. |
|
19
|
|
|
20
|
You'll need a device or emulator running **Android 8.0 (API 26) or higher**. An x86_64 emulator works fine, though a real ARM phone will feel noticeably faster. |
|
21
|
|
|
22
|
Hit Run. The first launch will download the model over your internet connection — after that the app is fully offline. |
|
23
|
|
|
24
|
--- |
|
25
|
|
|
26
|
## Project layout |
|
27
|
|
|
28
|
``` |
|
29
|
example/ |
|
30
|
├── settings.gradle.kts standalone project root |
|
31
|
├── build.gradle.kts deps and build config |
|
32
|
└── src/main/ |
|
33
|
├── AndroidManifest.xml INTERNET permission lives here |
|
34
|
└── kotlin/com/ondeinference/example/ |
|
35
|
├── MainActivity.kt one activity, sets up the theme and hands off to Compose |
|
36
|
├── ChatViewModel.kt model loading, streaming, history — all the logic |
|
37
|
└── ui/ |
|
38
|
├── ChatScreen.kt everything you see on screen |
|
39
|
└── theme/Theme.kt Material3 colours, dynamic on Android 12+ |
|
40
|
``` |
|
41
|
|
|
42
|
--- |
|
43
|
|
|
44
|
## How the streaming works |
|
45
|
|
|
46
|
When you send a message, the app drops an empty assistant bubble into the list immediately, then fills it in character by character as tokens arrive from the Rust engine. The ViewModel holds a `StringBuilder` and swaps the tail of the message list on every chunk — so Compose only recomposes the one bubble that's changing, not the whole conversation. |
|
47
|
|
|
48
|
```kotlin |
|
49
|
onde.stream(text).collect { chunk -> |
|
50
|
buffer.append(chunk.delta) |
|
51
|
// update only the last bubble in the list |
|
52
|
} |
|
53
|
``` |
|
54
|
|
|
55
|
--- |
|
56
|
|
|
57
|
## Swapping the model |
|
58
|
|
|
59
|
The app loads the platform default (Qwen 2.5 1.5B) via `loadDefaultModel()`. To use a different model, call `loadModel()` with a config instead: |
|
60
|
|
|
61
|
```kotlin |
|
62
|
onde.loadModel( |
|
63
|
config = OndeModels.qwen25_3b(), // ~1.93 GB — not great on phones |
|
64
|
systemPrompt = "You are a creative writing assistant.", |
|
65
|
sampling = OndeSampling.deterministic(), |
|
66
|
) |
|
67
|
``` |
|
68
|
|
|
69
|
Available sampling presets: |
|
70
|
|
|
71
|
| Preset | Temp | Max tokens | Good for | |
|
72
|
|---|---|---|---| |
|
73
|
| `OndeSampling.mobile()` | 0.7 | 128 | everyday chat | |
|
74
|
| `OndeSampling.default()` | 0.7 | 512 | longer responses | |
|
75
|
| `OndeSampling.deterministic()` | 0.0 | 512 | coding, facts | |
|
76
|
|
|
77
|
--- |
|
78
|
|
|
79
|
## Developing against a local SDK build |
|
80
|
|
|
81
|
By default the app pulls `onde-inference` from Maven Central. If you want to test changes to the SDK itself, open the parent directory (`onde/sdk/kotlin/`) in Android Studio instead and swap the dependency in `build.gradle.kts`: |
|
82
|
|
|
83
|
```kotlin |
|
84
|
// instead of: |
|
85
|
implementation("com.ondeinference:onde-inference:0.1.3") |
|
86
|
|
|
87
|
// use: |
|
88
|
implementation(project(":lib")) |
|
89
|
``` |
|
90
|
|
|
91
|
--- |
|
92
|
|
|
93
|
## Troubleshooting |
|
94
|
|
|
95
|
**Download never finishes** — check your internet connection. HuggingFace Hub occasionally rate-limits anonymous downloads; setting an `HF_TOKEN` env variable before building can help. |
|
96
|
|
|
97
|
**App crashes immediately** — make sure you're on API 26+. The SDK uses `Os.setenv` to configure the HuggingFace cache paths, which requires Android 8.0. |
|
98
|
|
|
99
|
**Inference is very slow** — CPU inference on an emulator is slow by design. Try a physical ARM device for a much better experience. |
|
100
|
|
|
101
|
**Out of memory on load** — the 1.5B model needs roughly 1–1.5 GB of free RAM. Close other apps and try again, or restart the device. |