External Publication
Visit Post

The BPE pre-tokenizer was not recognized!

Hugging Face Forums [Unofficial] May 5, 2026
Source

I believe the following should be added to the llama.cpp-convert_hf_to_gguf.py function:

def get_vocab_base_pre(self, tokenizer)

if chkhsh == “1444df51289cfa8063b96f0e62b1125440111bc79a52003ea14b6eac7016fd5f”:

ref: Qwen/Qwen3.5-4B-Base · Hugging Face

res = “qwen2”

Discussion in the ATmosphere

Loading comments...