{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreia4522n2yjpn444qg5lnwgt4bbkgm7rhswpowchfxmhwtziaodi3y",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mmog56vxupr2"
  },
  "path": "/t/sfttrainerflags-blocks-assistant-only-loss-true/176210#post_2",
  "publishedAt": "2026-05-25T10:22:56.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Qwen/Qwen3.5-4B-Base",
    "Qwen/Qwen3.5-4B-Base model card",
    "Qwen/Qwen3.5-4B-Base Transformers usage snippet",
    "huggingface/trl#5471 — Add {% generation %} chat templates for common model families",
    "TRL SFT Trainer docs — train on assistant messages only",
    "SFTConfig.assistant_only_loss",
    "Transformers tokenizer docs — return_assistant_tokens_mask",
    "Transformers chat templates",
    "Transformers multimodal chat templates",
    "Transformers processors",
    "huggingface/transformers#36713 — return_assistant_tokens_mask argument is blocked in ProcessorMixin.apply_chat_template",
    "huggingface/transformers#44521 — apply_chat_template returns all-zero assistant_masks for multimodal input",
    "huggingface/trl#3751 — VLM SFT computes loss for the whole sequence, including prompt/user content",
    "TRL SFT Trainer — dataset formats",
    "TRL SFT Trainer — train on completion only",
    "SFTConfig.completion_only_loss",
    "huggingface/trl#5471",
    "huggingface/trl#3751"
  ],
  "textContent": "It looks like this is falling into a gap between supported paths:\n\n* * *\n\nI think your read is basically right, but I would frame it slightly differently:\n\nThis does not look like a simple user mistake. It looks like a boundary case between:\n\n  1. a model artifact that is published/handled as a VLM,\n  2. a user workflow that is intentionally text-only,\n  3. TRL’s current `SFTTrainer` VLM path, and\n  4. the still-fragile implementation details behind `assistant_only_loss=True`.\n\n\n\nThe short version is:\n\n> `assistant_only_loss=True` is supported for the text/conversational SFT path when assistant-token masks can be produced from the chat template, but TRL currently blocks it for VLMs. Since `Qwen/Qwen3.5-4B-Base` resolves through the VLM/processor path, this check is triggered even if your dataset is text-only.\n\nThat makes the error understandable, but the UX is confusing.\n\n## Why this happens\n\nThe model page for Qwen/Qwen3.5-4B-Base is not presented like a plain text-only Causal LM path. The Hub usage panel shows `image-text-to-text`, `AutoProcessor`, and `AutoModelForImageTextToText` usage for this artifact:\n\n  * Qwen/Qwen3.5-4B-Base model card\n  * Qwen/Qwen3.5-4B-Base Transformers usage snippet\n\n\n\nSo even if your training data is text-only, the model artifact itself is still a vision-language capable artifact from the library/tooling point of view.\n\nOn the TRL side, the important current signal is this tracking issue:\n\n  * huggingface/trl#5471 — Add {% generation %} chat templates for common model families\n\n\n\nThat issue explicitly lists VLM families, including Qwen3.5, and notes:\n\n> VLMs currently don’t support `assistant_only_loss` in SFT (blocked by a separate check).\n\nSo I would not treat this as a random failure. It is a known unsupported combination.\n\n## Why `assistant_only_loss=True` is more fragile than it looks\n\n`assistant_only_loss=True` is not just a boolean that says “ignore user messages.” It depends on a whole chain working correctly:\n\n  * the dataset must be in the expected conversational format;\n  * the chat template must support assistant-token masking;\n  * the template needs `{% generation %}` / `{% endgeneration %}` markers;\n  * the tokenizer/processor must return a correct assistant mask;\n  * truncation must not remove the supervised assistant span;\n  * the data collator must actually apply that mask to `labels`;\n  * and in VLMs, image placeholders / image tokens / processor-specific expansion must not shift or break the mask.\n\n\n\nThe TRL docs describe the text/conversational path here:\n\n  * TRL SFT Trainer docs — train on assistant messages only\n  * SFTConfig.assistant_only_loss\n\n\n\nThe Transformers docs describe the lower-level assistant-mask mechanism here:\n\n  * Transformers tokenizer docs — return_assistant_tokens_mask\n  * Transformers chat templates\n  * Transformers multimodal chat templates\n  * Transformers processors\n\n\n\nThere are also related issues showing why this area is brittle, especially once `ProcessorMixin` / multimodal paths are involved:\n\n  * huggingface/transformers#36713 — return_assistant_tokens_mask argument is blocked in ProcessorMixin.apply_chat_template\n  * huggingface/transformers#44521 — apply_chat_template returns all-zero assistant_masks for multimodal input\n  * huggingface/trl#3751 — VLM SFT computes loss for the whole sequence, including prompt/user content\n\n\n\nThat is why the TRL check is conservative. Allowing `assistant_only_loss=True` on the VLM path without correct masking would be worse than raising an error, because it could silently train on the wrong tokens.\n\n## What I would try first for a strictly text-only experiment\n\nIf your workload is truly text-only, I would first try to force the text/tokenizer path by explicitly passing the tokenizer as `processing_class`.\n\nFor example:\n\n\n    from transformers import AutoTokenizer, AutoModelForCausalLM\n    from trl import SFTTrainer, SFTConfig\n\n    model_id = \"Qwen/Qwen3.5-4B-Base\"\n\n    tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n    model = AutoModelForCausalLM.from_pretrained(\n        model_id,\n        torch_dtype=\"auto\",\n        device_map=\"auto\",\n    )\n\n    trainer = SFTTrainer(\n        model=model,\n        args=SFTConfig(\n            output_dir=\"<your-output-dir>\",\n            assistant_only_loss=True,\n            # other args...\n        ),\n        train_dataset=train_dataset,\n        processing_class=tokenizer,\n    )\n\n\nThe important part is not just creating an `AutoTokenizer`. The important part is passing it into `SFTTrainer`:\n\n\n    processing_class=tokenizer\n\n\nOtherwise, `SFTTrainer` may resolve the default processing class through the processor path, and the model is then treated as a VLM.\n\nHowever, I would treat this as a workaround, not as proof that VLM + `assistant_only_loss` is officially supported.\n\nAfter doing this, I would always inspect the labels before training:\n\n\n    batch = trainer.data_collator.torch_call([train_dataset[0]])\n    labels = batch[\"labels\"][0]\n\n    print(\"supervised tokens:\", (labels != -100).sum().item())\n    print(tokenizer.decode(labels[labels != -100]))\n\n\nYou want the decoded supervised text to contain only the assistant answer span. If system/user/prompt text appears there, then the loss mask is not doing what you think it is doing.\n\nThis check is important because “the trainer runs” and “the correct tokens receive loss” are not the same thing.\n\n## What I would prefer for a more robust text-only setup\n\nIf the actual goal is:\n\n> train only on the answer/completion, not on the prompt/user/system text\n\nthen I would prefer a prompt-completion dataset and `completion_only_loss=True`, instead of relying on assistant masks from a chat template.\n\nThat path is conceptually simpler because the supervised boundary is explicit in the dataset schema:\n\n\n    {\n        \"prompt\": \"<|im_start|>system\\n...\\n<|im_end|>\\n<|im_start|>user\\n...\\n<|im_end|>\\n<|im_start|>assistant\\n\",\n        \"completion\": \"The assistant answer goes here.<|im_end|>\"\n    }\n\n\nThen configure SFT like this:\n\n\n    from trl import SFTTrainer, SFTConfig\n\n    trainer = SFTTrainer(\n        model=model,\n        args=SFTConfig(\n            output_dir=\"<your-output-dir>\",\n            completion_only_loss=True,\n            assistant_only_loss=False,\n        ),\n        train_dataset=prompt_completion_dataset,\n        processing_class=tokenizer,\n    )\n\n\nRelevant docs:\n\n  * TRL SFT Trainer — dataset formats\n  * TRL SFT Trainer — train on completion only\n  * SFTConfig.completion_only_loss\n\n\n\nFor this particular case, I would consider `completion_only_loss=True` the more durable route if you do not actually need multimodal inputs.\n\n## If you really want to train it as a VLM\n\nIf you are actually using images/multimodal messages, I would not currently expect `assistant_only_loss=True` to work through the normal TRL SFT path.\n\nThe relevant upstream signal is still:\n\n  * huggingface/trl#5471: VLMs currently do not support `assistant_only_loss` in SFT.\n  * huggingface/trl#3751: VLM SFT examples currently raise the question of masking out prompt/user content.\n\n\n\nIn that case, the choices are roughly:\n\n  1. accept whole-sequence loss for now;\n  2. restructure into prompt-completion if your data shape allows it;\n  3. write a custom collator and verify labels very carefully;\n  4. wait for upstream VLM assistant-mask support.\n\n\n\nA custom collator is possible, but I would be careful. For VLMs, masking is not just “find assistant text and set everything else to `-100`.” You also need to account for processor output, image placeholders, image tokens, multimodal token expansion, truncation, special tokens, and model-specific label handling.\n\n## What I think TRL could improve\n\nThe current error is technically reasonable:\n\n\n    Assistant-only loss is not yet supported for vision-language models.\n\n\nBut for this specific failure mode, it does not explain why a text-only user hit a VLM error.\n\nA more helpful error would say something like:\n\n\n    This model is being treated as a VLM because the resolved processing class is a ProcessorMixin/AutoProcessor path. VLMs do not currently support assistant_only_loss in SFT. If your workload is strictly text-only, try passing a tokenizer explicitly as processing_class=tokenizer, or use a prompt-completion dataset with completion_only_loss=True.\n\n\nThat would make the workaround and the distinction much clearer.\n\n## My practical recommendation\n\nFor a quick experiment:\n\n\n    processing_class=tokenizer\n\n\nthen inspect `labels != -100`.\n\nFor a more reliable text-only fine-tuning setup:\n\n\n    completion_only_loss=True\n    assistant_only_loss=False\n\n\nwith an explicit prompt-completion dataset.\n\nFor actual multimodal/VLM SFT:\n\n\n    assistant_only_loss=False\n\n\nunless you are prepared to write and validate a custom collator.\n\nSo overall: I think your intuition is right that this is a “gap” rather than a simple user error. But given the current TRL issue tracker and docs, I would treat the current behavior as an intentional unsupported-path guard, not as a straightforward bug.",
  "title": "SFTTrainerflags blocks assistant_only_loss=True"
}