Skip to content
Jorge Menjivar

Why tekken-rs became tekken

@jorge-menjivar

A few weeks ago I published tekken 0.2.0, a rename of the tekken-rs crate I've been maintaining. This week, vLLM merged a PR migrating to the new crate name, which felt like a nice milestone.

Why rename?

The short answer: the -rs suffix was initially added to show that this was a Rust implementation of the Mistral tokenizer spec and also to prevent any confusion if an official Tekken crate was ever published by Mistral AI themselves. That has not happened yet. Instead, big projects like vLLM, Candle, Sonar, and Huawei's KVarN have all adopted the tekken-rs crate and conveniently, rs is a conventional Rust suffix so it felt like a natural choice. But the library name inside the crate was always tekken. Every consumer wrote use tekken::... while their Cargo.toml said tekken-rs. That mismatch was a small but persistent source of confusion:

  • Searching for the crate on crates.io meant knowing the -rs suffix
  • The package = "tekken-rs" alias in downstream Cargo.toml files was a constant "why is this here?"
  • New contributors to projects using the crate would naturally try cargo add tekken and hit the wrong package

So a couple of weeks ago I decided to finally rename the crate to tekken on crates.io. Since crates.io has no rename mechanism, this meant publishing a new crate. tekken-rs 0.1.2 is the final release under the old name and will not receive further updates.

What 0.2.0 adds

The rename was the easy part. Version 0.2.0 also ships real feature work:

  • Tokenizer v15 support: the latest Mistral tokenizer spec
  • Image tokenization: tokenizing image inputs alongside text
  • Feature-gated audio and image dependencies: hound, rubato, rustfft, and ndarray are no longer pulled in unconditionally

That last point matters more than it sounds. In vLLM's case, the Rust workspace only uses tekken for text tokenization. With default-features = false now actually doing something, the audio dependency tree drops out entirely. No more carrying FFT libraries and audio decoders in a text-only pipeline.

The vLLM migration

The PR was almost trivially small, which is exactly the point of the rename:

  • No code changes needed. use tekken::... imports kept working because the library name was already tekken
  • The package = "tekken-rs" alias in the workspace dependency went away
  • default-features = false went from a no-op to actually trimming the dependency tree

All 1,555 workspace tests passed. @BugenZhao approved it and noted it was good to see no duplicated tiktoken versions in the dependency graph. Merged today.

If you're using tekken-rs

If your project depends on tekken-rs, the migration is a one-line change in Cargo.toml:

toml
# Before
tekken-rs = "0.1"

# After
tekken = "0.2"

No import changes, no API changes for text tokenization. Audio users don't need to change anything: the audio feature is enabled by default, matching the old crate where audio dependencies were unconditional. Image support is new in 0.2.0 and also enabled by default. If you only need text tokenization, default-features = false now actually drops the audio and image dependency trees.