{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreibolif47ky5xaobpl6vn3rto4kkefycoercd5ia6pzlcgynk35cp4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mmo7gku2gmo2"
  },
  "path": "/t/sfttrainerflags-blocks-assistant-only-loss-true/176210#post_1",
  "publishedAt": "2026-05-25T08:04:06.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "Hi. I came across this interesting case. I’m fine-tuning Qwen3.5-4B-Base for language-modeling only (no multi-model). I’m using AutoTokenizer instead of AutoProcessor. I updated the tokenizer_chat_template to add the {%generation%} tag so I can use assistant_only_loss=True\n\nWhen I set assistant_only_loss=True, I get the following objection from SFTTrainer:\n\n**ValueError: Assistant-only loss is not yet supported for vision-language models. Please set `assistant_only_loss=False` in the `SFTConfig`.**\n\nThe justification I got is: _“It seems like the error is coming from`SFTTrainer` itself — it inspects the model’s `config` to detect if it’s a VLM. Qwen3.5 has vision capabilities registered in its config, so even if you’re using it as a language-only model, `SFTTrainer`flags it as a VLM and blocks `assistant_only_loss=True`.”_\n\nI wonder if there’s a workaround for SFTTrainer to treat the model as language only since I’m defining it as: from transformers import AutoModelForCausalLM.\n\nThanks!",
  "title": "SFTTrainerflags blocks assistant_only_loss=True"
}