dart_sentencepiece_tokenizer
1.4.1, released 2 months ago by brodykim.work
A pure Dart implementation of SentencePiece tokenizer supporting BPE and Unigram.
What it does
It provides a pure Dart implementation of SentencePiece tokenizers supporting BPE (Gemma), Unigram (Llama), and Hugging Face tokenizer.json pipelines. It offers full encoding, decoding, batch processing, and a streaming API.
AI-written summary of version 1.4.1, based on the package's README and metadata. Report an inaccuracy
Use it when
- You need to tokenize text for LLMs like Llama or Gemma directly in pure Dart or Flutter.
- You want compatibility with Hugging Face tokenizer.json pipelines without relying on native bindings.
Skip it when
- You are using a non-SentencePiece based tokenizer that requires completely different architectures.
Strengths
- Pure Dart implementation with zero dependencies, working across all platforms.
- Memory efficient with typed arrays and optimized BPE merge operations.
- Includes support for Hugging Face tokenizer.json and a real-time text streaming API.
Weaknesses
- Input text exceeding 500,000 characters throws an ArgumentError to prevent out-of-memory errors.
Setup
- Kind
- Pure Dart
- Effort
- Low
- Code generation
- No
- Direct dependencies
- 0
import 'package:dart_sentencepiece_tokenizer/dart_sentencepiece_tokenizer.dart';
void main() {
final tokenizer = SentencePieceTokenizer.fromModelFileSync(
'tokenizer.model',
config: SentencePieceConfig.llama,
);
final encoding = tokenizer.encode('Hello, world!');
print(encoding.tokens);
print(encoding.ids);
}Alternatives
Packages that do the same job and could replace dart_sentencepiece_tokenizer.
- llamadart
Pick llamadart if you need a full Llama model execution environment rather than just a tokenizer.
93Compare - tflite_flutter
Pick tflite_flutter if you are running TensorFlow Lite models and require its specific ecosystem integrations.
68Compare