{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiddjmiijuc3uuotashsj6nrbej56c4yhdlumylyiprd77qyt23uui",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mp4gu7jwrlb2"
  },
  "path": "/t/i-built-a-novel-triple-hybrid-llm-mamba-attention-32-expert-moe-from-scratch-for-50-titan-v1-complete-titan-v2-first-cycle-done-expanding-dataset-now/177063#post_5",
  "publishedAt": "2026-06-25T10:51:45.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "**Quick clarification on the PPL numbers** — I think there’s a misread in the post. The PPL ~57.5 for Titan v2 isn’t a final result — it’s a snapshot from Epoch 2 of an ongoing run. Every model starting cold with a brand new 64K vocabulary begins around PPL 100+ (Titan v2 opened at PPL 87.87 at Epoch 0, which is actually quite clean for a cold start). The post was showing the trajectory, not the destination.\n\nHere’s where things actually stand as of today:\n\n**Cycle 1** (original dataset, ~1.8B tokens):\n\n_Epoch 0_ : PPL 87.87 ← cold start, new vocabulary\n\n_Epoch 1:_ PPL 61.37 ← −26.5 points in one epoch\n\n_Epoch 2:_ PPL 57.50 ← Cycle 1 complete\n\n**Cycle 2** (expanded dataset, 3.78B tokens/epoch):\n\n**Step 45,000** : val_loss 3.8319 → PPL ~46.1 ← current BEST\n\n**Step 47,000** : autoresume checkpoint — still training\n\n_Still actively converging. Target is ~9.3B total tokens (Chinchilla optimum for 464M params)_.\n\nOn all your technical points — you’re right across the board, and I’m taking each one seriously:\n\n**BPB instead of PPL** — absolutely, and this is now on my list before the paper submission. The v1 (50K vocab) vs v2 (64K vocab) comparison in raw PPL is misleading and I should have flagged it myself. BPB is the right cross-tokenizer metric and reviewers will ask for it.\n\n**Per-domain eval** — planned for the next checkpoint report. The 45/18/20/17 mix is a hypothesis I want to validate, not a result I’ve confirmed. Per-domain BPB across checkpoints will tell me whether FineWeb is carrying everything or whether Polish and bio are genuinely learning.\n\n**The ablation** — this is the one I keep circling back to. Mamba-only vs Attn+MoE-only vs triple-hybrid at matched params and tokens, run at 57M scale. It’s cheap and it’s the figure the architecture claim actually rests on. I’ll run it.\n\n**SFT as a controlled pair** — same SFT data, same recipe, two bases: 1.8B checkpoint vs the 9.3B one. That’s a clean figure and I’ll run it exactly that way.\n\n**On the Chinchilla framing** — noted. 9B is a checkpoint, not a ceiling. LLaMA proved empirically that for small models you train well past compute-optimal because inference is cheap and you’re buying capability, not minimizing FLOPs. If the loops persist past 9B, more tokens is the lever.\n\n**On the novelty** — fair and appreciated. Jamba is right there and I cite it. The defensible claim is the fixed-shape dispatch + loss-free EMA balancing + doing it from scratch under 1B for ~$50. Leading with the engineering contribution rather than the architectural combination is the stronger move for the paper abstract.\n\n​I’ll post a full checkpoint update this weekend—I’m taking some time off from work to focus entirely on BPB numbers and the per-domain breakdown. Would genuinely love your eyes on the Titan v2 model card if you have a moment — it has the full training config, architecture details, and I just updated it with the corrected training progress.\n\n_**This is the most useful single comment this project has received**_.\n\n_**Thank you!!!**_\n\nMati83moni",
  "title": "🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now"
}