Skip to content
pubscout

dart_sentencepiece_tokenizer

1.4.1, released 2 months ago by brodykim.work

A pure Dart implementation of SentencePiece tokenizer supporting BPE and Unigram.

$ flutter pub add dart_sentencepiece_tokenizer
Source repository

What it does

It provides a pure Dart implementation of SentencePiece tokenizers supporting BPE (Gemma), Unigram (Llama), and Hugging Face tokenizer.json pipelines. It offers full encoding, decoding, batch processing, and a streaming API.

AI-written summary of version 1.4.1, based on the package's README and metadata. Report an inaccuracy

Use it when

  • You need to tokenize text for LLMs like Llama or Gemma directly in pure Dart or Flutter.
  • You want compatibility with Hugging Face tokenizer.json pipelines without relying on native bindings.

Skip it when

  • You are using a non-SentencePiece based tokenizer that requires completely different architectures.

Strengths

  • Pure Dart implementation with zero dependencies, working across all platforms.
  • Memory efficient with typed arrays and optimized BPE merge operations.
  • Includes support for Hugging Face tokenizer.json and a real-time text streaming API.

Weaknesses

  • Input text exceeding 500,000 characters throws an ArgumentError to prevent out-of-memory errors.

Setup

Kind
Pure Dart
Effort
Low
Code generation
No
Direct dependencies
0
import 'package:dart_sentencepiece_tokenizer/dart_sentencepiece_tokenizer.dart';

void main() {
  final tokenizer = SentencePieceTokenizer.fromModelFileSync(
    'tokenizer.model',
    config: SentencePieceConfig.llama,
  );

  final encoding = tokenizer.encode('Hello, world!');
  print(encoding.tokens);
  print(encoding.ids);
}

Alternatives

Packages that do the same job and could replace dart_sentencepiece_tokenizer.

  • llamadart

    Pick llamadart if you need a full Llama model execution environment rather than just a tokenizer.

  • tflite_flutter

    Pick tflite_flutter if you are running TensorFlow Lite models and require its specific ecosystem integrations.