GPT 4.1 Character Encoding Issues?

Hi following your long discussions, I believe that it’s due to a restriction in OpenAI’s endpoint, which does not allow unicode escape sequences, except those for control characters.

See more with my original post:

GPT-4o returning malformed Unicode like \u0000e6 instead of æ — encoding bug? Bugs

We did an investigation and are confident that it is not from the tokenizer/tokenization. Instead, the endpoints seem to ban the generation of escaped Unicode sequence, other than control characters (0x0000-0x001f and 0x007f). But this is an undocumented behaviour, we still need confirmation from @OpenAI_Support @OpenAIAPIhelper Three things to work this around now: → Set ensure_ascii=False in your Python json.dumps() calls; → Remove \uXXXX examples from your prompts; → Avoid asking to gener…