onde_inference
1.4.0, released 5 days ago by ondeinference.com
Run Qwen 2.5 LLMs locally on device using Metal and CPU.
What it does
Runs on-device LLM inference for Flutter and Dart applications locally without using a cloud backend or API key. Supports streaming, multi-turn chat, sampling config, and custom UQFF models.
AI-written summary of version 1.3.5, based on the package's README and metadata. Report an inaccuracy
Use it when
- You need fully offline LLM inference inside a Flutter application.
- You want to run Qwen 2.5 models locally on iOS, macOS, Android, Windows, or Linux.
Skip it when
- You are targeting web browsers, which are not supported.
- You want a lightweight package without large native binaries and Rust toolchain requirements.
Strengths
- No cloud dependency, API keys, or data leaving the device.
- Supports hardware acceleration via Metal on iOS and macOS.
- Provides streaming and multi-turn chat capabilities.
Weaknesses
- Requires a working Rust toolchain and compiling large native dependency trees during build.
- CPU-bound inference on Android and desktop platforms can be slow.
Setup
- Kind
- Plugin (native code)
- Effort
- High
- Code generation
- Yes (build_runner)
- Direct dependencies
- 2
Requires a working Rust toolchain, flutter_rust_bridge setup, and cargo-ndk for Android builds.
Future<void> main() async {
WidgetsFlutterBinding.ensureInitialized();
await OndeInference.init();
final engine = OndeChatEngine();
await engine.loadDefaultModel(systemPrompt: 'You are a helpful assistant.');
final result = await engine.sendMessage(message: 'What is Rust ownership?');
print(result.text);
}Alternatives
Packages that do the same job and could replace onde_inference.
- llamadart
Alternative library for running local LLMs in Dart.
93Compare - tflite_flutter
Alternative for running TensorFlow Lite models on-device.
68Compare
Works with
Packages commonly used together with onde_inference: ones it builds on, ones built on it, and its own family.