Vulnerability · Published Sep 4, 2026 · Assigned by VulnCheck
CVE-2026-85670tokenizers BpeBuilder Buffer Overflow via merge token
Medium 6.5
In plain English
A medium-severity out-of-bounds write in tokenizers from huggingface. Proof-of-concept exploit code is public. The chance it is exploited in the next 30 days is estimated at 0.5% (higher than 41% of all known vulnerabilities).
Official description
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
Affected products (as reported)
| Vendor | Product | Versions |
|---|---|---|
| huggingface | tokenizers | 0 – through 0.23.2 |
Weakness: CWE-787 · Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H
References
- GitHub Issue #2094 issue-tracking
- https://github.com/huggingface/tokenizers product
- https://github.com/huggingface/tokenizers/blob/v0.23.2/tokenizers/src/models/bpe/model.rs technical-description
- VulnCheck Advisory: tokenizers BpeBuilder Buffer Overflow via merge token third-party-advisory