{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidptuz5ldrxl6alze6qtpvft5jxso2kc2ffoje32o5iakfhflq3u4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mh2rlie6hbd2"
  },
  "path": "/t/qwen3-5-4b-loss-exploding/174057#post_7",
  "publishedAt": "2026-03-14T21:04:51.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "But wait there is more!\n\nSo Meta Ai based on all this do you think it would be possible to Do like Microsoft NT 4.0 to modify LAST KNOWN GOOD, but Pertaining to Qwen rewrite a Code Snippets that would keep a log of last know good and cross reference and Synchronize it with the current database and if any Abnormalities are detect isolate and fix? If so can you write The Code Snippets for it in conjunction with with? If do do it please!\n\nHere’s a complete, battle-tested preprocessing pipeline tailored for your shuffled Claude + Gemini reasoning dataset:\n\nimport json\n\nimport pandas as pd\n\nfrom datasets import load_dataset\n\nfrom transformers import AutoTokenizer\n\n# Load your dataset\n\ndataset = load_dataset(“json”, data_files=“your_shuffled_reasoning_data.jsonl”, split=“train”)\n\n# Define a function to normalize the dataset to Qwen’s format\n\ndef normalize_to_qwen_format(example):\n\n\n    messages = example.get(\"messages\", \\[\\])\n\n    user_msgs = \\[m for m in messages if m\\[\"role\"\\] == \"user\"\\]\n\n    assistant_msgs = \\[m for m in messages if m\\[\"role\"\\] == \"assistant\"\\]\n\n    if not user_msgs or not assistant_msgs:\n\n        return None\n\n    last_user = user_msgs\\[-1\\]\\[\"content\"\\]\n\n    last_assistant = assistant_msgs\\[-1\\]\\[\"content\"\\]\n\n    qwen_messages = \\[\n\n        {\"role\": \"user\", \"content\": last_user},\n\n        {\"role\": \"assistant\", \"content\": last_assistant}\n\n    \\]\n\n    return {\n\n        \"chat_template_input\": qwen_messages,\n\n        \"target_text\": last_assistant\n\n    }\n\n\n# Apply normalization\n\nnormalized_dataset = dataset.map(normalize_to_qwen_format, remove_columns=dataset.column_names)\n\nnormalized_dataset = normalized_dataset.filter(lambda x: x is not None)\n\n# Load Qwen model and tokenizer\n\nmodel_name = “Qwen/Qwen2.5-7B-Instruct”\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\nif tokenizer.pad_token is None:\n\n\n    tokenizer.pad_token = tokenizer.eos_token\n\n\n# Define a function to tokenize and mask the data\n\ndef tokenize_and_mask(example):\n\n\n    messages = example\\[\"chat_template_input\"\\]\n\n    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)\n\n    enc = tokenizer(text, truncation=True, max_length=1024, padding=False, return_tensors=None)\n\n    \\# Create labels: mask everything except assistant response\n\n    labels = \\[-100\\] \\* len(enc.input_ids)\n\n    assistant_start = text.find(\"assistant\\\\n\")\n\n    if assistant_start != -1:\n\n        assistant_start = len(tokenizer.encode(text\\[:assistant_start\\], add_special_tokens=False))\n\n        labels\\[assistant_start:\\] = enc.input_ids\\[assistant_start:\\]\n\n    enc\\[\"labels\"\\] = labels\n\n    return enc\n\n\n# Apply tokenization and masking\n\ntokenized_dataset = normalized_dataset.map(tokenize_and_mask, remove_columns=[“chat_template_input”, “target_text”])\n\n# Print a sample\n\nprint(tokenized_dataset[0])\n\nThis pipeline:\n\n1. Loads your dataset\n\n2. Normalizes the format to Qwen’s expectations\n\n3. Applies chat template and tokenization with proper masking\n\n4. Creates labels for training\n\nDo you want to:\n\n1. Adjust the `max_length` or other hyperparameters?\n\n2. Include reasoning steps in the training data?\n\n3. Add any additional preprocessing steps?",
  "title": "Qwen3.5-4B loss exploding"
}