tiktoken
1.0.3, released 4 years ago by marinvitlov.com
A Dart BPE tokeniser for OpenAI models to encode and decode text.
WASM-ready
What it does
Provides functions to encode text into token IDs and decode token IDs back to text. Includes model‑specific encodings and utilities to count tokens for API calls.
AI-written summary of version 1.0.3, based on the package's README and metadata. Report an inaccuracy
Use it when
- You need to count tokens for OpenAI API usage
- You want to encode/decode text with the same tokenisation as OpenAI models
- You need a lightweight, pure‑Dart solution without native dependencies
Skip it when
- You require a full‑featured LLM client rather than just tokenisation
- You need GPU‑accelerated tokenisation or Rust‑level performance
Strengths
- Pure Dart, works on all Flutter platforms
- No native dependencies, small footprint
- Supports multiple OpenAI model encodings
Weaknesses
- Only handles tokenisation, no higher‑level API client
- May be slower than native Rust/Python implementations
Setup
- Kind
- Pure Dart
- Effort
- Low
- Code generation
- No
- Direct dependencies
- 0
import 'package:tiktoken/tiktoken.dart';
final enc = encodingForModel('gpt-4');
print(enc.encode('tiktoken is great!')); // [83,1609,5963,374,2294,0]
print(enc.decode([83,1609,5963,374,2294,0])); // 'tiktoken is great!'Alternatives
Packages that do the same job and could replace tiktoken.