{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiaxe6avqon3dwcxadc2byzeus4vqzyqkolp5cqufqgwdyqjrvsa24",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mn6wzlyvdxv2"
  },
  "path": "/t/gemma4-e4b-adaptors-fuse-after-training-how/176429#post_1",
  "publishedAt": "2026-05-31T23:58:14.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "i manage to do it with qwen3.5, Gemma-4-e2b but after many attempts AI assisted i don’t get anything but errors, more than one million tokens with deep seek, many scripts, even retrained the model with mlx-vlm and mlx-lm . My las adapters where produced with this: cmd = [\nVENV, “-m”, “mlx_vlm.lora”,\n“–model-path”, MODEL,\n“–dataset”, DATA,\n“–batch-size”, “1”,\n“–iters”, “300”,\n“–learning-rate”, “1e-5”,\n“–grad-checkpoint”,\n“–gradient-accumulation-steps”, “4”,\n“–steps-per-save”, “50”,\n“–output-path”, OUTPUT,\n“–lora-rank”, “8”,\n“–lora-alpha”, “16”,\n“–train-on-completions”,\n“–assistant-id”, “4368”,\n]\n\nand this is the fusion script :\n#!/usr/bin/env python3\n“”“Fusion LoRA adapters into Gemma 4 E4B 8-bit model.\nEjecutar DIRECTAMENTE en el Mini, no por SSH.\n“””\nimport json, shutil\nfrom pathlib import Path\nimport mlx.core as mx\nimport mlx.nn as nn\nfrom mlx_vlm import load\nfrom mlx_vlm.trainer.lora import LoRaLayer\n\nBASE = Path(“/Users/hal9000/Desktop/AI/modelos/gemma_4_e4b_it_8bit”)\nADAPTER = Path(“/Volumes/ssd./ssd_gemma4/adaptadores_v2”)\nSALIDA = Path(“/Users/hal9000/Desktop/AI/modelos/gemma_4_e4b_it_8bit_ssd_fused”)\n\n# adapter_config.json\n\nwith open(ADAPTER / “adapter_config.json”, “w”) as f:\njson.dump({“rank”: 8, “alpha”: 16.0, “dropout”: 0.0}, f)\n\nprint(“Cargando modelo con adapters…”)\nmodel, processor = load(str(BASE), adapter_path=str(ADAPTER))\n\nprint(“Fusionando capas LoRA…”)\nto_update = {}\nfor name, module in model.named_modules():\nif not isinstance(module, LoRaLayer):\ncontinue\norig = module.original_layer\nif isinstance(orig, nn.QuantizedLinear):\nw = mx.dequantize(orig.weight, orig.scales, orig.biases, orig.group_size, orig.bits)\nelse:\nw = orig.weight\nlu = module.scale * (module.A @ module.B)\nod, id_ = w.shape\nif lu.shape == (id_, od):\nwf = w + lu.T\nelif lu.shape == (od, id_):\nwf = w + lu\nelse:\nprint(\" SKIP shape mismatch:“, name, w.shape, lu.shape)\ncontinue\nto_update[name] = wf\nprint(” %d capas fusionadas\" % len(to_update))\n\nprint(“Cargando pesos originales…”)\nall_w = {}\nfor sf in sorted(BASE.glob(“model-*.safetensors”)):\nall_w.update(mx.load(str(sf)))\nprint(\" %d tensores totales\" % len(all_w))\n\n# Reemplazar pesos fusionados\n\nfor name, wf in to_update.items():\nmk = name + “.weight”\nif mk in all_w:\nall_w[mk] = wf\nfor s in [“.scales”, “.biases”]:\nall_w.pop(name + s, None)\n\n# VERIFICAR shapes conv contra originales y restaurar si cambiaron\n\nprint(“Verificando shapes conv…”)\nrestored = 0\nfor sf in sorted(BASE.glob(“model-*.safetensors”)):\norig = mx.load(str(sf))\nfor k, v in orig.items():\nif len(v.shape) == 4 and k in all_w and all_w[k].shape != v.shape:\nprint(\" RESTAURANDO %s: %s → %s\" % (k, list(all_w[k].shape), list(v.shape)))\nall_w[k] = v\nrestored += 1\nprint(\" %d conv weights restaurados\" % restored)\n\n# Guardar\n\nif SALIDA.exists():\nshutil.rmtree(SALIDA)\nSALIDA.mkdir(parents=True)\n\nprint(“Guardando %d tensores…” % len(all_w))\nmx.save_safetensors(str(SALIDA / “model.safetensors”), all_w)\n\n# Copiar configs\n\nfor f in BASE.glob(“*”):\nif f.suffix in (“.json”, “.txt”, “.md”) or “tokenizer” in f.name:\nshutil.copy2(f, SALIDA)\n\n# Eliminar index.json\n\nidx = SALIDA / “model.safetensors.index.json”\nif idx.exists():\nidx.unlink()\n\nprint(\"OK \" + str(SALIDA))\n\n# Verificacion final\n\nprint(“Verificando conv weights en output…”)\nd = mx.load(str(SALIDA / “model.safetensors”))\nok = True\nfor k, v in d.items():\nif “conv” in k.lower() and len(v.shape) == 4:\nprint(\" %s: %s\" % (k, list(v.shape)))\n\n# Verificar contra original\n\nfor sf in sorted(BASE.glob(“model-*.safetensors”)):\norig = mx.load(str(sf))\nif k in orig and v.shape != orig[k].shape:\nprint(\" ERROR: deberia ser %s\" % list(orig[k].shape))\nok = False\nif ok:\nprint(“TODOS los conv weights tienen shapes correctos!”)",
  "title": "Gemma4-e4b adaptors fuse after training , how?"
}