SFTTrainerflags blocks assistant_only_loss=True

Hi. I came across this interesting case. I’m fine-tuning Qwen3.5-4B-Base for language-modeling only (no multi-model). I’m using AutoTokenizer instead of AutoProcessor. I updated the tokenizer_chat_template to add the {%generation%} tag so I can use assistant_only_loss=True

When I set assistant_only_loss=True, I get the following objection from SFTTrainer:

ValueError: Assistant-only loss is not yet supported for vision-language models. Please set assistant_only_loss=False in the SFTConfig.

The justification I got is: “It seems like the error is coming fromSFTTrainer itself — it inspects the model’s config to detect if it’s a VLM. Qwen3.5 has vision capabilities registered in its config, so even if you’re using it as a language-only model, SFTTrainerflags it as a VLM and blocks assistant_only_loss=True.”

I wonder if there’s a workaround for SFTTrainer to treat the model as language only since I’m defining it as: from transformers import AutoModelForCausalLM.

Thanks!