Skip to content
pubscout

onde_inference

1.4.0, released 5 days ago by ondeinference.com

Run Qwen 2.5 LLMs locally on device using Metal and CPU.

$ flutter pub add onde_inference
Source repository

What it does

Runs on-device LLM inference for Flutter and Dart applications locally without using a cloud backend or API key. Supports streaming, multi-turn chat, sampling config, and custom UQFF models.

AI-written summary of version 1.3.5, based on the package's README and metadata. Report an inaccuracy

Use it when

  • You need fully offline LLM inference inside a Flutter application.
  • You want to run Qwen 2.5 models locally on iOS, macOS, Android, Windows, or Linux.

Skip it when

  • You are targeting web browsers, which are not supported.
  • You want a lightweight package without large native binaries and Rust toolchain requirements.

Strengths

  • No cloud dependency, API keys, or data leaving the device.
  • Supports hardware acceleration via Metal on iOS and macOS.
  • Provides streaming and multi-turn chat capabilities.

Weaknesses

  • Requires a working Rust toolchain and compiling large native dependency trees during build.
  • CPU-bound inference on Android and desktop platforms can be slow.

Setup

Kind
Plugin (native code)
Effort
High
Code generation
Yes (build_runner)
Direct dependencies
2

Requires a working Rust toolchain, flutter_rust_bridge setup, and cargo-ndk for Android builds.

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await OndeInference.init();
  final engine = OndeChatEngine();
  await engine.loadDefaultModel(systemPrompt: 'You are a helpful assistant.');
  final result = await engine.sendMessage(message: 'What is Rust ownership?');
  print(result.text);
}

Alternatives

Packages that do the same job and could replace onde_inference.

Works with

Packages commonly used together with onde_inference: ones it builds on, ones built on it, and its own family.