[Feldt]

BpeTokenizer::Decode can return invalid UTF-8 for runs of byte-fallback tokens (<0xHH>). FlushBytes takes the sequence length from the lead byte alone, although its comment says invalid sequences become U+FFFD:

• a continuation byte (0x80-0xBF) or 0xF8-0xFF on its own is appended raw: [<0xA0>] decodes to "\xA0";
• a lead byte is appended with whatever bytes follow it, unchecked: [<0xC2>, <0xC2>] decodes to "\xC2\xC2";
• a truncated sequence gives one U+FFFD and then its raw continuation bytes: [<0xE2>, <0x80>] decodes to "\xEF\xBF\xBD\x80".

SentencePiece decodes each of these to one U+FFFD per byte. Such runs reach Decode when the model samples a malformed byte sequence, and when a byte-fallback character is decoded one token at a time, as run.cc and api_server.cc do while streaming. api_server.cc then passes the text to json::dump(), which throws on invalid UTF-8 by default.

Change

• New ValidUtf8Len() returns the length of the well-formed UTF-8 sequence at a position (lead C2-DF, E0-EF or F0-F4, continuation bytes 10xxxxxx, no overlong, surrogate or above-U+10FFFF code points), or 0.
• FlushBytes() keeps well-formed sequences and emits U+FFFD for every other byte; its comment now says so. The encode-side Utf8Len() / DecodeUtf8() are untouched.

Scope:

• Decode on the byte-fallback path now always returns valid UTF-8, matching SentencePiece.
• It does not recover a character whose bytes arrive in separate Decode calls: "€" (<0xE2>, <0x82>, <0xAC>) decoded one token at a time was lost before (U+FFFD, raw \x82, raw \xAC) and still is (three U+FFFD). That needs buffering in the caller, as deepseek/run_dsv4.cc does with CompleteUtf8Prefix().
• DecodeByteLevel (Qwen3-style tokenizer.json) is unchanged, because its raw bytes let per-token output concatenate back into a character split across byte-level tokens, and HF-style U+FFFD would break that. Streaming api_server.cc needs caller-side buffering there, which is a separate change.

Question: SentencePiece or HF rule?

The old comment said "matching HuggingFace ByteFallback". HF replaces the whole run with U+FFFD if any part is invalid; SentencePiece keeps the well-formed sequences. So [<0xC3>, <0xA9>, <0xA9>] gives three U+FFFD in HF tokenizers 0.23.3, but "é" + U+FFFD in SentencePiece and here. This PR follows SentencePiece because bpe_tokenizer_test.cc already uses it as this backend's reference, and a Gemma checkpoint can be converted from either tokenizer.model or tokenizer.json, so Decode output stays the same for both. Switching to the HF rule is a small change if you prefer it.

Testing

New test BpeTokenizerTest.DecodeMalformedByteFallbackMatchesSentencePiece, next to DecodeRoundTrip, checks valid, lone continuation, invalid lead, missing continuation, truncated, overlong, surrogate, above-U+10FFFF and mixed runs against fixed expectations and SentencePiece, then compares the two on all 65,792 runs of one or two byte tokens. On dev it fails (10 listed cases, 35,720 runs differ); with the fix, all 8 tests in bpe_tokenizer_test pass.

bazel test //:bpe_tokenizer_test cannot be analyzed in the OSS tree because :packed_tokenizer needs the undeclared //python:convert_from_safetensors, so the test was built with CMake against the pinned highway/sentencepiece:

./bpe_tokenizer_test
./bpe_tokenizer_test --gtest_filter=BpeTokenizerTest.DecodeMalformedByteFallbackMatchesSentencePiece

Also checked: on 50,000 random 1-8 token sequences, this code and the sentencepiece 0.2.2 Python package differ 38,665 times on dev (all invalid UTF-8) and 0 times with this change, also under ASan/UBSan.

This change was prepared with an AI coding agent (Claude Code), which also ran the checks above. The submitter is responsible for it and will answer review.

🤖 Generated with Claude Code

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.