Skip to content
pubscout

tiktoken

1.0.3, released 4 years ago by marinvitlov.com

A Dart BPE tokeniser for OpenAI models to encode and decode text.

WASM-ready
$ flutter pub add tiktoken
Source repository

What it does

Provides functions to encode text into token IDs and decode token IDs back to text. Includes model‑specific encodings and utilities to count tokens for API calls.

AI-written summary of version 1.0.3, based on the package's README and metadata. Report an inaccuracy

Use it when

  • You need to count tokens for OpenAI API usage
  • You want to encode/decode text with the same tokenisation as OpenAI models
  • You need a lightweight, pure‑Dart solution without native dependencies

Skip it when

  • You require a full‑featured LLM client rather than just tokenisation
  • You need GPU‑accelerated tokenisation or Rust‑level performance

Strengths

  • Pure Dart, works on all Flutter platforms
  • No native dependencies, small footprint
  • Supports multiple OpenAI model encodings

Weaknesses

  • Only handles tokenisation, no higher‑level API client
  • May be slower than native Rust/Python implementations

Setup

Kind
Pure Dart
Effort
Low
Code generation
No
Direct dependencies
0
import 'package:tiktoken/tiktoken.dart';
final enc = encodingForModel('gpt-4');
print(enc.encode('tiktoken is great!')); // [83,1609,5963,374,2294,0]
print(enc.decode([83,1609,5963,374,2294,0])); // 'tiktoken is great!'

Alternatives

Packages that do the same job and could replace tiktoken.