Onde Inference

Run LLMs on-device from Flutter and Dart. Metal on iOS and macOS, CPU everywhere else.

Rust SDK · Swift SDK · Kotlin Multiplatform SDK · React Native SDK · Website

---

Run Qwen 2.5 models directly inside your Flutter app. The model downloads from Hugging Face on first launch, then everything runs locally. No server, no API key, and no user data leaving the device. On an iPhone 15 Pro, Metal gets you around 15 tok/s. Android and desktop run on CPU, so they are slower, but they work well for local inference.

You get multi-turn chat, streaming, one-shot generation, and configurable sampling in a single package.

#Platform support

Platform	Backend	Default model	Notes
iOS 13+	Metal	Qwen 2.5 1.5B (~941 MB)	Simulator uses aarch64-apple-ios-sim
macOS 10.15+	Metal	Qwen 2.5 3B (~1.93 GB)	Apple silicon and Intel
Android API 21+	CPU	Qwen 2.5 1.5B (~941 MB)	arm64-v8a, armeabi-v7a, x86_64, x86
Linux x86_64	CPU	Qwen 2.5 3B (~1.93 GB)	CUDA possible, see docs
Windows x86_64	CPU	Qwen 2.5 3B (~1.93 GB)	CUDA possible, see docs

Web is not supported. On-device inference needs native system access that browsers don't expose.

---

#Quick start

dependencies:
  onde_inference: ^0.1.0

The inference engine is written in Rust and wired into Dart through flutter_rust_bridge. You will need a working Rust toolchain. The first build is usually slow, around 5 to 10 minutes, because it has to compile the full dependency tree. After that, incremental builds are much faster.

#Initialize

Call this once at startup, before anything else:

import 'package:onde_inference/onde_inference.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await OndeInference.init();
  runApp(const MyApp());
}

#Load a model

final engine = await OndeChatEngine.create();

// Picks the default model for the device:
//   iOS / Android → Qwen 2.5 1.5B (~941 MB)
//   macOS / Linux / Windows → Qwen 2.5 3B (~1.93 GB)
final elapsed = await engine.loadDefaultModel(
  systemPrompt: 'You are a helpful assistant.',
);
print('Model loaded in ${elapsed.toStringAsFixed(1)} s');

#Chat

final result = await engine.sendMessage('What is Rust's ownership model?');
print(result.text);
print('Generated in ${result.durationDisplay}');

#Stream

final buffer = StringBuffer();

await for (final chunk in engine.streamMessage('Tell me a short story.')) {
  buffer.write(chunk.delta);
  setState(() => _displayText = buffer.toString());
  if (chunk.done) break;
}

#Engine status

final info = await engine.info();

print(info.status);        // EngineStatus.ready
print(info.modelName);     // "Qwen 2.5 3B"
print(info.approxMemory);  // "~1.93 GB"
print(info.historyLength); // number of turns so far

#History

final history = await engine.history();
for (final msg in history) {
  print('${msg.role}: ${msg.content}');
}

// Clear history but keep the model loaded.
final removed = await engine.clearHistory();
print('Cleared $removed messages.');

// Seed from a saved session — no inference runs.
await engine.pushHistory(ChatMessage.user('Hello from last session!'));
await engine.pushHistory(ChatMessage.assistant('Hi! How can I help today?'));

#One-shot generation

This runs inference without changing conversation history. It is useful for things like prompt enhancement, classification, or formatting.

final result = await engine.generate(
  [
    ChatMessage.system('You are a JSON formatter. Output only valid JSON.'),
    ChatMessage.user('Name: Alice, Age: 30, City: Stockholm'),
  ],
  sampling: SamplingConfig.deterministic(),
);
print(result.text);

#Unload

await engine.unloadModel();

---

#Model selection

// Platform-aware default (recommended).
final config = OndeInference.defaultModelConfig();

// Force a specific model.
final small  = OndeInference.qwen251_5bConfig();   // ~941 MB
final medium = OndeInference.qwen253bConfig();      // ~1.93 GB
final coder  = OndeInference.qwen25Coder3bConfig(); // ~1.93 GB, code-tuned

await engine.loadGgufModel(
  medium,
  systemPrompt: 'You are an expert software engineer.',
);

Model	Size	Good for
Qwen 2.5 1.5B Instruct Q4_K_M	~941 MB	iOS, tvOS, Android
Qwen 2.5 3B Instruct Q4_K_M	~1.93 GB	macOS, Linux, Windows
Qwen 2.5 Coder 1.5B Instruct Q4_K_M	~941 MB	Code on mobile
Qwen 2.5 Coder 3B Instruct Q4_K_M	~1.93 GB	Code on desktop

---

#Sampling

All fields are optional. null means "use the engine default".

final sampling = SamplingConfig(
  temperature: 0.7,
  topP: 0.95,
  topK: 40,
  maxTokens: 256,
);

await engine.setSampling(sampling);

Presets:

SamplingConfig.defaultConfig()   // temp=0.7, max 512 tokens
SamplingConfig.deterministic()   // greedy, temp=0.0
SamplingConfig.mobile()          // temp=0.7, max 128 tokens

---

#Error handling

All engine methods throw OndeException on failure:

try {
  await engine.loadDefaultModel();
} on OndeException catch (e) {
  debugPrint('Inference error: ${e.message}');
}

Common causes include calling sendMessage before loading a model, having no internet connection on first run while the model still needs to download, or running out of memory. The 3B model needs about 2 GB free, so on tighter devices you will usually want the 1.5B model instead.

---

#Sandboxed app setup (iOS / macOS)

On iOS and sandboxed macOS, the default Hugging Face cache path sits outside the app container. Call setupCacheDir() once at startup so Onde uses a location your app can actually access:

import 'package:onde_inference/onde_inference.dart';
import 'package:path_provider/path_provider.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await OndeInference.init();

  String? fallback;
  if (Platform.isIOS || Platform.isAndroid) {
    final dir = await getApplicationSupportDirectory();
    fallback = dir.path;
  }
  await OndeInference.setupCacheDir(fallbackDir: fallback);

  runApp(const MyApp());
}

Onde first tries the App Group shared container (group.com.ondeinference.apps) so Onde-powered apps can share downloaded models. If that App Group is not configured, it falls back to the app's private directory.

---

#Contributing

The source lives at github.com/ondeinference/onde:

• Rust core: src/
• Dart bridge crate: sdk/dart/rust/
• Dart library: sdk/dart/lib/
• Example app: sdk/dart/example/

Open an issue before sending large PRs.

#License

Onde is dual-licensed under MIT and Apache 2.0. You can use either one.

• MIT License
• Apache License 2.0

© 2026 Splitfire AB

---

#Copyright

© 2026 Onde Inference (Splitfire AB).