@ondeinference / onde
Code Commits Issues Pulls Cloud Agent CI/CD
development
copilot/assess-inference-engine-complexity copilot/fix-onde-npm-sdk-v1-1-2-release development feature/diffusion-vision main
md 6.58 KB
View source Raw

Onde Inference

Onde Inference

Run LLMs on-device from React Native with Onde Inference. Metal on iOS, CPU on Android, and no user data leaving the device.

npm Website License

Rust SDK · Swift SDK · Kotlin Multiplatform SDK · Flutter SDK · Website

Run Qwen 2.5 models directly on the device. No server, no API key, and no user data leaving the phone. For an efficient on-device inference engine for React Native, the SDK page is the quickest place to check install details and platform notes. If you want to verify model downloads or GGUF output before you ship the app build, use Onde CLI.

The model downloads from Hugging Face the first time you load it, then runs locally after that. The 1.5B model is about 941 MB. On iPhone, Metal makes it feel surprisingly fast. On Android, it runs on CPU, so it is slower, but it still works well enough for local chat.

Installation

npx expo install @ondeinference/react-native

Quick start

import { OndeChatEngine, userMessage } from "@ondeinference/react-native";

// Picks the default model for the device:
//   iOS     → Qwen 2.5 1.5B (~941 MB, Metal)
//   Android → Qwen 2.5 1.5B (~941 MB, CPU)
const seconds = await OndeChatEngine.loadDefaultModel(
  "You are a helpful assistant."
);

const reply = await OndeChatEngine.sendMessage("Hello!");
console.log(reply.text);

// One-shot — doesn't touch conversation history
const expanded = await OndeChatEngine.generate(
  [userMessage("Expand: a cat in space")],
  { temperature: 0.0 }
);

await OndeChatEngine.unloadModel();

Platforms

Platform Backend Default model
iOS Metal Qwen 2.5 1.5B (~941 MB)
Android CPU Qwen 2.5 1.5B (~941 MB)

API

OndeChatEngine

Method Returns What it does
loadDefaultModel(systemPrompt?, sampling?) Promise<number> Load the platform default. Returns load time in seconds.
loadModel(config, systemPrompt?, sampling?) Promise<number> Load a specific GGUF model.
loadUqffModel(config, systemPrompt?, sampling?) Promise<number> Load a specific UQFF model.
unloadModel() `Promise<string \ null>`
isLoaded() boolean Is anything loaded right now?
info() Promise<EngineInfo> Status, model name, memory, history length.
sendMessage(message) Promise<InferenceResult> Chat turn. Appends to history automatically.
generate(messages, sampling?) Promise<InferenceResult> One-shot. History stays untouched.
setSystemPrompt(prompt) void Replace the system prompt.
clearSystemPrompt() void Remove it.
setSampling(config) void Swap sampling params.
history() Promise<ChatMessage[]> Full conversation so far.
clearHistory() number Wipe it. Returns how many messages were removed.
pushHistory(message) void Inject a message without running inference.

Helpers

import {
  defaultModelConfig,     // platform-aware (1.5B on mobile, 3B on desktop)
  uqffModelConfig,        // base repo/local dir + UQFF shard(s)
  qwen251_5bConfig,       // force 1.5B (~941 MB)
  qwen253bConfig,         // force 3B (~1.93 GB)
  defaultSamplingConfig,  // temp=0.7, top_p=0.95, max_tokens=512
  deterministicSamplingConfig,  // temp=0.0
  mobileSamplingConfig,   // temp=0.7, max_tokens=128
  systemMessage,
  userMessage,
  assistantMessage,
} from "@ondeinference/react-native";

UQFF example:

const config = uqffModelConfig(
  "google/gemma-4-E4B-it",
  ["q4k-0.uqff"],
  "Gemma 4 E4B (UQFF Q4K)",
  "~2.5 GB (UQFF Q4K)",
);

await OndeChatEngine.loadUqffModel(config);

Example app

There is a working chat app in example/:

cd example
npm install
npx expo run:ios

It is a single-file example, about 290 lines, and it covers loading, chat, status, history management, and error handling.

Building from source

You need Rust and the right cross-compilation targets.

# iOS
rustup target add aarch64-apple-ios aarch64-apple-ios-sim
./scripts/build-rust.sh ios

# Android (set ANDROID_NDK_HOME first)
rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android i686-linux-android
./scripts/build-rust.sh android

The script builds the Rust FFI bridge in rust/, then copies the static library for iOS or the shared libraries for Android into the right places under ios/ and android/.

How it fits together

TypeScript  →  Expo Module (Swift / Kotlin)  →  Rust C FFI  →  onde crate  →  mistral.rs
                @_silgen_name (iOS)                               ↓
                JNI external (Android)                     Metal / CPU

The native module talks to Rust through extern "C" functions. Complex types cross the boundary as JSON strings, and the TypeScript layer handles camelCase ↔ snake_case conversion. A global tokio::Runtime, created once, runs the async inference work.

License

Onde is dual-licensed under MIT and Apache 2.0. You can use either one.

© 2026 Splitfire AB (Onde Inference).