{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifwbicatwfmhj7kao2mf7x6dq5nhmgx5klrqgjgyu5ypme73v3zii",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mouogumuid52"
  },
  "path": "/t/i-built-a-novel-triple-hybrid-llm-mamba-attention-32-expert-moe-from-scratch-for-50-titan-v1-complete-titan-v2-first-cycle-done-expanding-dataset-now/177063#post_1",
  "publishedAt": "2026-06-22T09:49:03.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Mati83moni/HybridMoE-Titan-v1 · Hugging Face",
    "Mati83moni/HybridMoE-Titan-v2 · Hugging Face",
    "HybridMoE Titan - a Hugging Face Space by Mati83moni"
  ],
  "textContent": "_**Hi HuggingFace community**_\n\nI’m Mateusz — independent ML researcher from UK, operating under **Project Inkblot**. By day I work as a Manufacturing Controller, this project is something I’ve been building in my spare time, driven purely by passion and curiosity. Today I want to share something I’ve been working on for months: a full pre-trained language model designed, implemented, and trained entirely from scratch, combining Mamba SSM, Multi-Head Attention, and fine-grained 32-expert MoE in a single decoder-only architecture — to my knowledge the first to do so **under 1B parameters**.\n\n* * *\n\n##  What is HybridMoE Titan v1?\n\nA **450M parameter decoder-only language model** that unifies three fundamentally different sequence processing paradigms in a single architecture:\n\nComponent | Role\n---|---\n**Mamba SSM** (12 layers) | Linear-time local context, O(T) compute\n**Multi-Head Attention + RoPE** (4 layers) | Global context capture\n**Fine-Grained MoE** | 32 routed + 2 shared experts, top-2 routing\n\nThis is **not a wrapper, fine-tune, or modification** of any existing model. Every component — architecture, training loop, data pipeline, checkpointing — written from scratch in PyTorch.\n\n* * *\n\n##  What’s actually novel here\n\n### 1. Fixed-Shape Expert Dispatch\n\nEvery standard MoE uses `torch.nonzero()` which creates variable-length tensors incompatible with gradient checkpointing. Titan solves this with a fixed-shape dispatch that works with full gradient checkpointing across all 16 layers:\n\n\n    VRAM with gradient checkpointing: 5.84 GB\n    Without:                         >24 GB\n\n\n\nThis is the difference between requiring an A100 cluster and training on a **single L4 GPU**.\n\n### 2. Loss-Free EMA Load Balancing\n\nStandard MoE (Switch, Mixtral) adds auxiliary balance loss that competes with the LM objective. Titan uses EMA-tracked router logit biases — **zero auxiliary loss, zero competing gradients** :\n\n\n    Dead experts:     0 / 32  (entire training run)\n    usage_std:        0.019 – 0.031  (ideal uniform = 0.031)\n    Max expert load:  14.7%\n\n\n\nTextbook routing health. No dead experts, no collapse.\n\n### 3. Jamba-Style 1:3 Layer Interleaving\n\n\n    Layer  0:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]\n    Layer  1:  [Mamba SSM]      + [SwiGLU Dense FFN]\n    Layer  2:  [Mamba SSM]      + [SwiGLU Dense FFN]\n    Layer  3:  [Mamba SSM]      + [SwiGLU Dense FFN]\n    Layer  4:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]\n    ...\n\n\n\nMamba handles local patterns in O(T). Attention captures global context every 4 layers. MoE provides specialized capacity exactly where global reasoning happens.\n\n### 4. Single-Line Scalability\n\nThe entire model is parameterized through one config. Scaling from 57M to 1.3B+ parameters requires changing **one integer** :\n\n\n    config = HybridModelConfig(num_layers=8)   # Titan Tiny    57.7M\n    config = HybridModelConfig(num_layers=16)  # Titan v1     450.4M\n    config = HybridModelConfig(num_layers=24)  # Titan Mid    690.0M\n    config = HybridModelConfig(num_layers=48)  # Titan Large  ~1.3B\n\n\n\nThe 1:3 interleaving, expert counts, RoPE — everything propagates automatically.\n\n* * *\n\n##  Training Results (Titan v1)\n\nConfig | Value\n---|---\nHardware | Single NVIDIA L4 24GB\nInstance | AWS g6.2xlarge\nPrecision | bfloat16\nEffective batch | 64 sequences (~65K tokens)\nPeak throughput | ~6,200 tok/s\nTotal tokens trained | ~4.2B\n**Total cost** | **~$50**\n\n### Validation Perplexity Progression\n\nStep | Val Loss | PPL | Notes\n---|---|---|---\n12,000 | 2.38 | 10.81 | 50-shard val set\n20,000 | 3.47 | 32.31 |  Expanded to 200 shards (Polish added)\n26,000 | 3.38 | 29.26 | Monotonic improvement resumes\n42,850 | 3.30 | ~27.5 | **Final checkpoint**\n\n> The PPL increase at step 20K is from validation set expansion (50→200 shards, adding Polish text) — not model degradation. Within the 200-shard regime loss decreases monotonically.\n\n* * *\n\n##  Titan v2 — First Training Cycle Complete\n\nTitan v2 completed its **first full pre-training cycle (3 epochs)** with major improvements across every component:\n\nAspect | v1 | v2\n---|---|---\nParameters | 450.4M | **464.5M**\nTokenizer | GPT-2 BPE 50K | **Custom BPE 64K** (PL + EN + bio + code)\nPolish support | Fragmented (ą→2 tokens) | **Native single-token**\nOptimizer | AdamW | **AdamW(Tested and Stable)**\nTraining | Single GPU | **Single GPU** + **gradient checkpointing**\nData pipeline | Runtime loading | **Pre-tokenized binary shards**\nData quality | Unknown | **Full corruption audit — 0 binary artifacts**\n\n### Titan v2 Pre-Training — Completed Cycle\n\n\n    Start:          Fresh init, custom 64K tokenizer\n    Epoch 0:        PPL  87.87  ← cold start, new vocabulary\n    Epoch 1:        PPL  61.37  ← −26.5 points in one epoch\n    Epoch 2:        PPL  57.50  ← best_val_loss = 4.0523  ✅ FINAL\n\n    Total tokens:   ~1.8B  (3 epochs × ~0.6B per epoch)\n    Total steps:    27,981\n    Hardware:       AWS g6.2xlarge (L4 24GB) + benchmarking on A100\n    Duration:       6 days  (Jun 11 → Jun 17 2026)\n    Cost:           ~$40\n\n\n\nFour-domain corpus: 45% FineWeb-Edu (English) + 18% Polish (articles + Wolne Lektury) + 20% biomedical (PubMed/clinical) + 17% Python/code.\n\n**Currently:** Building an expanded ~4.56B token dataset for the next training cycle, targeting **9B total tokens** — Chinchilla-optimal scale for 464M parameters.\n\n* * *\n\n##  Links\n\n  * **Titan v1 — Model + Weights + Paper:** Mati83moni/HybridMoE-Titan-v1 · Hugging Face\n  * **Titan v2 — Live training:** Mati83moni/HybridMoE-Titan-v2 · Hugging Face\n  * **Demo Space:** HybridMoE Titan - a Hugging Face Space by Mati83moni\n  * **DOI:** Mati83moni/HybridMoE-Titan-v1 · Hugging Face\n\n\n\nThe model weights, full architecture code, training scripts, and paper are all public on HuggingFace. **Fully open source, MIT license.**\n\n* * *\n\n##  Looking for arXiv Endorsement\n\nThe paper is pending submission to **arXiv cs.LG**. As a first-time independent submitter I need **one endorsement** from an established arXiv author in cs.LG or cs.AI.\n\nIf you’re endorsed for cs.LG and find this work interesting — please reach out via HuggingFace messages. One endorsement is all I need.\n\nThe paper covers: architecture design, fixed-shape dispatch, EMA load balancing, training dynamics, and scaling analysis.\n\n* * *\n\n##  Happy to discuss\n\n  * Why 1:3 interleaving over 1:1 or 1:7?\n  * Fixed-shape dispatch implementation details\n  * Training stability on a single GPU without loss spikes\n  * The $50 compute budget — what’s possible and what isn’t\n  * Roadmap to Titan Standard (~1B) and beyond\n\n\n\nBuilt from scratch, one GPU, one researcher. **Project Inkblot**",
  "title": "🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now"
}