@ondeinference / onde
Code Commits Issues Pulls Cloud Agent CI/CD
feature/diffusion-vision
copilot/assess-inference-engine-complexity copilot/fix-onde-npm-sdk-v1-1-2-release development feature/diffusion-vision main
md 225 lines 6.81 KB
Rendered Raw
1 <p align="center">
2 <h1 align="center">onde-inference</h1>
3 <p align="center"><strong>On-device AI inference for Python</strong></p>
4 </p>
5
6 <p align="center">
7 <a href="https://pypi.org/project/onde-inference/"><img src="https://img.shields.io/pypi/v/onde-inference.svg" alt="PyPI version"></a>
8 <a href="https://pypi.org/project/onde-inference/"><img src="https://img.shields.io/pypi/pyversions/onde-inference.svg" alt="Python 3.9+"></a>
9 <a href="#license"><img src="https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg" alt="License"></a>
10 </p>
11
12 ---
13
14 Run LLMs and speech-to-text locally — no cloud, no latency, no data leaving the device. Powered by Rust via [mistral.rs](https://github.com/EricLBuehler/mistral.rs).
15
16 `onde-inference` is a Python binding for the [Onde](https://github.com/ondeinference/onde) Rust crate. It wraps the Rust inference engine via [UniFFI](https://mozilla.github.io/uniffi-rs/) bindings, giving you native-speed on-device inference with a simple, async Python API.
17
18 ## Installation
19
20 ```bash
21 pip install onde-inference
22 ```
23
24 ## Quick Start
25
26 ### Basic Chat
27
28 ```python
29 import asyncio
30 from onde_inference import OndeChatEngine, default_model_config
31
32
33 async def main():
34 # Create the engine (starts with no model loaded).
35 engine = OndeChatEngine()
36
37 # Load the platform-appropriate default model.
38 # macOS → Qwen 2.5 3B (~1.93 GB), Linux/Windows → Qwen 2.5 3B (~1.93 GB)
39 elapsed = await engine.load_default_model(
40 system_prompt="You are a helpful assistant.",
41 sampling=None,
42 )
43 print(f"Model loaded in {elapsed:.1f}s")
44
45 # Send a message — history is managed automatically.
46 result = await engine.send_message("What is Rust?")
47 print(result.text)
48 print(f"Generated in {result.duration_display}")
49
50 # Multi-turn conversation.
51 follow_up = await engine.send_message("How does its ownership model work?")
52 print(follow_up.text)
53
54 # Cleanup.
55 await engine.unload_model()
56
57
58 asyncio.run(main())
59 ```
60
61 ### Model Configuration Helpers
62
63 Choose a specific model instead of the platform default:
64
65 ```python
66 from onde_inference import default_model_config, qwen25_1_5b_config, qwen25_3b_config
67
68 # Platform-appropriate default (3B on desktop, 1.5B on mobile).
69 config = default_model_config()
70
71 # Explicitly pick a model.
72 small = qwen25_1_5b_config() # Qwen 2.5 1.5B — ~941 MB
73 medium = qwen25_3b_config() # Qwen 2.5 3B — ~1.93 GB
74
75 # Load a specific model config.
76 elapsed = await engine.load_gguf_model(
77 config=small,
78 system_prompt="You are a helpful assistant.",
79 sampling=None,
80 )
81 ```
82
83 ### Sampling Configuration
84
85 Control how the model generates text:
86
87 ```python
88 from onde_inference import (
89 default_sampling_config,
90 deterministic_sampling_config,
91 mobile_sampling_config,
92 )
93
94 # Creative chat (temperature=0.7, top_p=0.95, max_tokens=512).
95 creative = default_sampling_config()
96
97 # Greedy decoding (temperature=0.0) — deterministic output.
98 greedy = deterministic_sampling_config()
99
100 # Conservative settings for constrained devices (max_tokens=128).
101 mobile = mobile_sampling_config()
102
103 # Pass to load or update at any time.
104 await engine.load_default_model(system_prompt="You are helpful.", sampling=greedy)
105 await engine.set_sampling(creative)
106 ```
107
108 ### Streaming
109
110 Stream tokens as they are generated using a callback class that implements `StreamChunkListener`:
111
112 ```python
113 import asyncio
114 from onde_inference import (
115 OndeChatEngine,
116 StreamChunkListener,
117 StreamChunk,
118 stream_chat_message,
119 )
120
121
122 class MyStreamHandler(StreamChunkListener):
123 def on_chunk(self, chunk: StreamChunk) -> bool:
124 """Called for each token. Return True to continue, False to cancel."""
125 if chunk.delta:
126 print(chunk.delta, end="", flush=True)
127 if chunk.done:
128 print() # newline at the end
129 return True
130
131
132 async def main():
133 engine = OndeChatEngine()
134 await engine.load_default_model(
135 system_prompt="You are a helpful assistant.",
136 sampling=None,
137 )
138
139 # stream_chat_message is a free function (UniFFI requirement).
140 await stream_chat_message(
141 engine=engine,
142 message="Tell me a short story about a robot learning to paint.",
143 listener=MyStreamHandler(),
144 )
145
146 await engine.unload_model()
147
148
149 asyncio.run(main())
150 ```
151
152 ### Message Helpers
153
154 Build message lists for one-shot generation without modifying conversation history:
155
156 ```python
157 from onde_inference import system_message, user_message, assistant_message
158
159 messages = [
160 system_message("You expand short prompts into detailed descriptions."),
161 user_message("a cat in space"),
162 ]
163
164 result = await engine.generate(messages=messages, sampling=None)
165 print(result.text)
166 ```
167
168 ### Engine Introspection
169
170 ```python
171 # Check engine state.
172 info = await engine.info()
173 print(info.status) # "ready", "unloaded", "generating", etc.
174 print(info.model_name) # "Qwen 2.5 3B" or None
175 print(info.history_length) # number of conversation turns
176
177 # Access or clear history.
178 history = await engine.history()
179 cleared = await engine.clear_history()
180
181 # Check if a model is loaded.
182 if await engine.is_loaded():
183 print("Ready to chat!")
184 ```
185
186 ## Platform Support
187
188 | Platform | Accelerator | Notes |
189 |----------|-------------|-------|
190 | **macOS** | Metal GPU | Apple Silicon and Intel Macs |
191 | **Linux** | CPU / CUDA | NVIDIA GPU acceleration with CUDA builds |
192 | **Windows** | CPU / CUDA | NVIDIA GPU acceleration with CUDA builds |
193
194 ## Supported Models
195
196 | Model | Size | Best For |
197 |-------|------|----------|
198 | **Qwen 2.5 1.5B** (GGUF Q4_K_M) | ~941 MB | Lightweight / memory-constrained environments |
199 | **Qwen 2.5 3B** (GGUF Q4_K_M) | ~1.93 GB | Balanced quality and performance on desktop |
200
201 Models are automatically downloaded from HuggingFace Hub on first use and cached locally.
202
203 ## How It Works
204
205 `onde-inference` is a thin Python layer over the **Onde** Rust crate. [UniFFI](https://mozilla.github.io/uniffi-rs/) generates Python bindings from the Rust source, so every call goes directly into native compiled code — there is no Python inference loop. Async methods use Python's `asyncio` (UniFFI generates async Python wrappers for Rust async functions backed by Tokio).
206
207 ## Links
208
209 - **Source**: [github.com/ondeinference/onde](https://github.com/ondeinference/onde)
210 - **Rust crate**: [crates.io/crates/onde](https://crates.io/crates/onde)
211 - **Documentation**: [docs.rs/onde](https://docs.rs/onde)
212 - **Inference engine**: [mistral.rs](https://github.com/EricLBuehler/mistral.rs)
213
214 ## License
215
216 Licensed under either of:
217
218 - Apache License, Version 2.0 ([LICENSE-APACHE](https://github.com/ondeinference/onde/blob/main/LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
219 - MIT License ([LICENSE-MIT](https://github.com/ondeinference/onde/blob/main/LICENSE-MIT) or <http://opensource.org/licenses/MIT>)
220
221 at your option.
222
223 ## Copyright
224
225 Copyright 2026 [Onde Inference](https://ondeinference.com)