{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreia5q5xetitj5t3557u6kyi4l2i5tckole4qfajnz4ytdgrglcygfu",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mphfiolc6vm2"
  },
  "path": "/t/i-built-a-novel-triple-hybrid-llm-mamba-attention-32-expert-moe-from-scratch-for-50-titan-v1-complete-titan-v2-first-cycle-done-expanding-dataset-now/177063#post_17",
  "publishedAt": "2026-06-29T20:21:58.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "@ProtopiaUk",
    "@KnackAU",
    "@vl-dev",
    "@John6666",
    "@torch.no_grad"
  ],
  "textContent": "#  Weekend Research Update — SFT Experiments, BPB Metric & New Pretraining BEST\n\n**Project Inkblot | Mateusz Piesiak | 26–28 June 2026**\n\n> _A quick personal note before diving in — I genuinely wish I had more time to participate actively in the discussion here. Life has been quite full lately: day job as a Manufacturing Controller, family, kids, the household, and everything else that comes with being a solo researcher doing this entirely on the side. I read every comment carefully and they matter a lot to me — I just do not always get to respond as quickly as I would like. This weekend report is my way of making up for the silence. Thank you for your patience._\n\n##  Pretraining Progress — New BEST Checkpoint\n\nSince my last post the pretraining has continued on AWS g6.2xlarge (L4 24GB). Here is the full checkpoint history:\n\nStep | Val Loss | PPL | Notes\n---|---|---|---\n30,000 | 4.0508 | 57.5 | Cycle 2 start (expanded dataset)\n45,000 | 3.8319 | 46.1 | Previous BEST\n50,000 | 3.7447 | 42.3 |\n53,950 | 3.7447 | 42.3 | Autoresume STOP\n55,000 | 3.6835 | 39.8 |\n60,000 | 3.6361 | 37.9 |\n**65,000** | **3.5968** | **~36.5** | **New BEST**\n\nThe model continues to converge steadily. At step 65,000 we have processed approximately **4.26B tokens** (65,000 steps × batch 4 × grad_accum 16 × seq_len 1024). Chinchilla optimum for 464M params is ~9.3B tokens so we are at **46% of the way**.\n\n##  Weekend SFT Experiments\n\nThis weekend I ran three SFT experiments. Here is exactly what happened, including the raw outputs.\n\n### Experiment 1 — Colab T4, 120 examples, freeze 10/16 layers\n\nI started with the step 50,000 checkpoint (PPL 42.3) and a small dataset of 120 ChatML examples across 5 domains. Froze the embedding and layers 0–9, training only the top 6 layers. Used fp16 with gradient checkpointing on a T4 (15GB VRAM), 7 epochs at LR 3e-6.\n\nThe training curve looked promising on paper:\n\n\n    Ep 1/7  val=7.49  PPL=1796\n    Ep 2/7  val=5.79  PPL=326\n    Ep 3/7  val=4.57  PPL=96\n    Ep 4/7  val=3.77  PPL=43\n    Ep 5/7  val=3.35  PPL=28\n    Ep 6/7  val=3.15  PPL=23\n    Ep 7/7  val=3.07  PPL=21.6  ← saved\n\n\nBut the actual generation told a very different story:\n\n\n    Prompt: \"Co to jest szlak mTOR i jaką rolę odgrywa w nowotworach?\"\n\n    Output: 123466678910101011121314151718181819206070909090899090909190909094909090\n            CCC C C C G G G H G G G M G G G A G G G I G G G I G G G I G G G I...\n\n\nPPL 21.6 after 7 epochs was pure memorisation, not learning. The model reproduced numerical patterns (TPM values 1, 2, 3… 100, 200, 500, 1000) from the RNA-seq training examples. 120 examples × 7 epochs = 742 forward passes on the same data. Classic overfitting on a tiny dataset.\n\n### Experiment 2 — Colab T4, 120 examples, full unfreeze\n\nSame 120 examples but this time all 16 layers unfrozen, 2 epochs.\n\n\n    Ep 1/2  val=7.49  PPL=1796\n    Ep 2/2  val=2.68  PPL=14.5  ← saved (interrupted)\n\n\nOutput:\n\n\n    Prompt: \"Explain the mechanism of action of tyrosine kinase inhibitors in CML\"\n\n    Output: /. / W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W\n            W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W...\n\n\nCatastrophic forgetting in 88 seconds. Training all 464M parameters on 120 examples erased 5.2B tokens of pretraining knowledge. The model replaced everything with a loop on a single token.\n\n### Experiment 3 — AWS L4, 4300 examples, structural MoE-aware freeze\n\nBased on KnackAU’s earlier comment and my own architecture inspection, I moved from sequential layer freezing to **structural freezing** — freezing by component type rather than by layer index:\n\n\n    for name, param in model.named_parameters():\n        freeze = (\n            'token_emb' in name or\n            any(f'layers.{i}.' in name for i in range(4)) or\n            'router' in name or\n            'routed_experts' in name or\n            'shared_experts' in name or\n            'shared_gate' in name\n        )\n        param.requires_grad = not freeze\n\n\nThe key decision here was freezing the MoE routers and experts. Why? Because the Expert Sparsity Analysis (done earlier this weekend) showed that the router learned domain-specific expert allocation with zero supervision:\n\n\n    Medical vs Python   overlap = 0.00  ← perfect domain specialisation\n    Medical vs Polish   overlap = 0.15  ← clear specialisation\n    Python  vs English  overlap = 0.11  ← clear specialisation\n\n\nThis emerged purely from pretraining on the mixed 4-domain corpus. SFT should not destroy this.\n\nThe split ended up at 278.6M frozen (embedding + first 4 layers + all MoE routing and experts) and 185.9M trainable (Attention QKV, Mamba SSM, Dense/SwiGLU, lm_head) — about 40% of the model.\n\nThe dataset was much larger this time: 4,300 ChatML examples total. 3,500 clinical reasoning examples with step-by-step analysis, 800 examples from MedDialog/MedQA/PubMedQA/ChatDoctor/iCliniq, and 50 of my own Polish medical examples covering RNA-seq omics and drug mechanisms.\n\nTraining curve:\n\n\n    Val BEFORE  9.92   PPL=20,257\n    Ep 1/3      train=0.46  val=0.060  PPL=1.1\n    Ep 2/3      train=0.04  val=0.042  PPL=1.0  ← BEST saved\n\n\nVal loss 0.04 looked incredible but it was an artefact of **data leakage**. When building the dataset I mixed train_final.csv and val.csv into one pool and then split 95/5 randomly. The model was being evaluated on examples it had already seen during training. The actual generalisation is unknown.\n\nThe output confirmed the problem:\n\n\n    Prompt: \"What is the mTOR signaling pathway and its role in cancer?\"\n\n    Output: Guthrie struct char ******* Tibpectral vagus neutrophonsai Monaster\n            sappveston CoQ provinandelion oracle alp powiatowszeostridochastic...\n\n\nRoot cause is the same as Experiments 1 and 2 — base model PPL 39.8 is simply too weak for SFT to produce anything useful. This confirms KnackAU’s point that Chinchilla is a checkpoint, not a ceiling. The SFT pipeline, dataset, and freeze strategy are all correct. The base model just needs more pretraining first.\n\n##  Evaluation Results (step 50,000, before SFT)\n\n### MCQ Log-Likelihood\n\nI ran a proper MCQ eval with full-text scoring and a BPE fix (the previous version had a merge bug where single-word continuations produced cont=0 tokens — fixed by stripping trailing space from prompt and including it with the continuation, which is the standard lm-evaluation-harness approach).\n\n\n    🟢 Easy    (5 questions)  3/5 (60%)  Paris ✅  print() ✅  Warszawa ✅\n    🟡 Medium  (5 questions)  2/5 (40%)  mTOR ✅  RNA-seq ✅\n    🔴 Hard    (5 questions)  0/5 (0%)   expected for base model pre-SFT\n\n\n### Tokenizer Compression — Titan 64K vs Mistral-7B vs GPT-2\n\n\n    Polish General    Titan 4.04 c/t  |  Mistral 2.47 (+67%)  |  GPT-2 1.93 (+110%)\n    Polish Medical    Titan 3.25 c/t  |  Mistral 2.30 (+41%)  |  GPT-2 2.07 (+57%)\n    English Tech      Titan 5.19 c/t  |  Mistral 4.84 (+7%)   |  GPT-2 5.05 (+3%)\n    Python Code       Titan 2.52 c/t  |  Mistral 2.77 (-9%)   |  GPT-2 2.22 (+14%)\n\n\nThe 64K Polish-optimised tokenizer gives +67% better compression for Polish text vs Mistral. This directly reduces compute cost for Polish-language inference.\n\n### Expert Sparsity — Router Specialisation across 4 MoE layers (L0/L4/L8/L12)\n\n\n    Medical (top-5)  E24:24  E29:23  E16:21  E23:21  E3:21\n    Python  (top-5)  E9:52   E28:50  E30:48  E27:47  E10:47\n    Polish  (top-5)  E24:26  E4:25   E30:23  E29:21  E22:21\n    English (top-5)  E23:18  E3:18   E30:15  E13:14  E29:14\n\n    Domain specialisation (IoU of top-5 expert sets)\n      Medical  vs Python   overlap = 0.00  ✅ Perfect specialisation\n      Medical  vs Polish   overlap = 0.15  ✅ Clear specialisation\n      Python   vs English  overlap = 0.11  ✅ Clear specialisation\n      Polish   vs English  overlap = 0.25  ✅ Clear specialisation\n\n\nThe router learned domain-specific expert allocation with zero supervision — this emerged purely from pretraining on the mixed 4-domain corpus.\n\n##  BPB Evaluator — responding to KnackAU’s feedback\n\nAs KnackAU correctly pointed out, PPL is tokenizer-dependent and cannot fairly compare Titan v1 (50K vocab, PPL ~27.5) with Titan v2 (64K vocab, PPL ~36.5).\n\n**Bits Per Byte (BPB)** is tokenizer-invariant:\n\n\n    BPB = total_NLL_nats / (ln(2) × n_bytes_UTF8)\n        = log₂(PPL) / avg_bytes_per_token\n\n\nImplementation:\n\n\n    @torch.no_grad()\n    def compute_bpb(model, tok, text: str) -> dict:\n        n_bytes  = len(text.encode('utf-8'))\n        ids      = tok.encode(text, add_special_tokens=False)\n\n        input_ids = torch.tensor([ids]).to(DEVICE)\n        out = model(input_ids)\n        logits = (out[0] if isinstance(out, tuple) else out)[0]\n\n        log_probs = F.log_softmax(logits[:-1].float(), dim=-1)\n        targets   = input_ids[0, 1:]\n        nll_per_token = -log_probs[range(len(targets)), targets]\n\n        total_nll = nll_per_token.sum().item()\n        bpb = total_nll / (math.log(2) * n_bytes)\n        return {\"bpb\": bpb, \"ppl\": math.exp(nll_per_token.mean().item())}\n\n\nThe evaluator tests across 5 domains (English General, Polish General, Medical English, Medical Polish, Python Code). BPB results for v1 vs v2 comparison will be published once v2 reaches the target PPL.\n\nInterpretation scale:\n\n\n    < 1.0    excellent (better than human for English)\n    1.0–1.5  good\n    1.5–2.0  average\n    > 2.0    undertrained\n\n\n##  SFT Dataset — Ready for Next Run\n\nThe dataset for the next SFT attempt is built and waiting:\n\n**sft_master_v2.jsonl** — 4,300 high-quality ChatML examples (7.6MB)\n\n  * 3,500 medical examples from train_final.csv — clinical reasoning with step-by-step analysis, system prompt “You are an expert medical AI assistant”\n  * 800 examples from val.csv — MedDialog, MedQA, PubMedQA, ChatDoctor, iCliniq, system prompt “You are an empathetic AI health companion”\n  * 50 of my own Polish medical examples — RNA-seq omics analysis, drug mechanism explanations (PL/EN), patient education in Polish\n\n\n\nValidation split is 197 examples in a **separate** file (sft_val_v2.jsonl) with no mixing with train. Lesson learned.\n\n##  SFT Architecture for Next Run\n\nFor the next SFT run (once pretraining reaches PPL ~30-32) the plan is the same structural freeze that was validated in Experiment 3:\n\n\n    # Structural freeze — preserves MoE domain specialisation\n    freeze = (\n        'token_emb' in name or\n        any(f'layers.{i}.' in name for i in range(4)) or\n        'router' in name or\n        'routed_experts' in name or\n        'shared_experts' in name or\n        'shared_gate' in name\n    )\n    # Frozen    278.6M (embedding + 4 layers + all MoE routing)\n    # Trainable 185.9M (Attention, Mamba, Dense, lm_head)\n\n\nTarget config is LR 3e-6, 3 epochs, GRAD_ACCUM 4, fp32 on L4.\n\n##  What’s Next\n\n\n    Current   step 65,000  |  PPL 36.5  |  4.26B tokens\n    Target    step ~92,000  |  PPL ~33-34  |  ~6B tokens (early stop considered)\n              step ~164,490  |  PPL ~30-32  |  9.3B tokens (Chinchilla optimum)\n\n\nAfter pretraining reaches the target:\n\n  1. BPB eval — v1 vs v2 cross-tokenizer comparison\n  2. SFT with sft_master_v2.jsonl (4,300 examples, structural freeze)\n  3. Per-domain eval post-SFT\n  4. Architecture ablation at 57M scale (Mamba-only vs Attn+MoE vs triple hybrid)\n\n\n\n##  Key Lessons This Weekend\n\n  1. **SFT requires a competent base.** PPL > 35 means garbage output regardless of dataset quality or freeze strategy. This is not a bug — it is physics.\n\n  2. **MoE expert routing is precious.** Freeze routers and experts during SFT. The domain specialisation (Medical vs Python overlap = 0.00) took billions of tokens to learn and can be destroyed in minutes.\n\n  3. **Data leakage is subtle.** Mixing sources before splitting produces optimistically low val loss that means nothing. Val set must be built separately before any train data is assembled.\n\n  4. **BPB > PPL for cross-model comparison.** Larger vocab mechanically inflates PPL. The models are more comparable than the raw PPL numbers suggest.\n*Update comming soon\n\n  5. **120 examples is not enough for 464M params.** Minimum viable SFT for this model size is ~500-1000 high-quality instruction pairs. The 4,300-example dataset should work.\n\n\n\n\nSo yee that happen on weekend guys have nice reading hehe  and yee\nAny Question Just Message me Happy to Answer For all Question\n@ProtopiaUk @KnackAU @vl-dev @John6666",
  "title": "🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now"
}