GPT-4o returning malformed Unicode like \u0000e6 instead of æ — encoding bug?

We did an investigation and are confident that it is not from the tokenizer/tokenization. Instead, the endpoints seem to ban the generation of escaped Unicode sequence, other than control characters (0x0000-0x001f and 0x007f). But this is an undocumented behaviour, we still need confirmation from @OpenAI_Support @OpenAIAPIhelper

Three things to work this around now: → Set ensure_ascii=False in your Python json.dumps() calls; → Remove \uXXXX examples from your prompts; → Avoid asking to generate unicode escapes in OpenAI or Azure OpenAI’s endpoints. UTF-8 representation in JSON is sufficient for the expressivity.

More details in our post: https://research.giskard.ai/blog/structured-output