External Publication

🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now

Hugging Face Forums [Unofficial] June 22, 2026

Hi HuggingFace community

I’m Mateusz — independent ML researcher from UK, operating under Project Inkblot. By day I work as a Manufacturing Controller, this project is something I’ve been building in my spare time, driven purely by passion and curiosity. Today I want to share something I’ve been working on for months: a full pre-trained language model designed, implemented, and trained entirely from scratch, combining Mamba SSM, Multi-Head Attention, and fine-grained 32-expert MoE in a single decoder-only architecture — to my knowledge the first to do so under 1B parameters.

What is HybridMoE Titan v1?

A 450M parameter decoder-only language model that unifies three fundamentally different sequence processing paradigms in a single architecture:

Component	Role
Mamba SSM (12 layers)	Linear-time local context, O(T) compute
Multi-Head Attention + RoPE (4 layers)	Global context capture
Fine-Grained MoE	32 routed + 2 shared experts, top-2 routing

This is not a wrapper, fine-tune, or modification of any existing model. Every component — architecture, training loop, data pipeline, checkpointing — written from scratch in PyTorch.

What’s actually novel here

1. Fixed-Shape Expert Dispatch

Every standard MoE uses torch.nonzero() which creates variable-length tensors incompatible with gradient checkpointing. Titan solves this with a fixed-shape dispatch that works with full gradient checkpointing across all 16 layers:

VRAM with gradient checkpointing: 5.84 GB
Without:                         >24 GB

This is the difference between requiring an A100 cluster and training on a single L4 GPU.

2. Loss-Free EMA Load Balancing

Standard MoE (Switch, Mixtral) adds auxiliary balance loss that competes with the LM objective. Titan uses EMA-tracked router logit biases — zero auxiliary loss, zero competing gradients :

Dead experts:     0 / 32  (entire training run)
usage_std:        0.019 – 0.031  (ideal uniform = 0.031)
Max expert load:  14.7%

Textbook routing health. No dead experts, no collapse.

3. Jamba-Style 1:3 Layer Interleaving

Layer  0:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]
Layer  1:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  2:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  3:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  4:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]
...

Mamba handles local patterns in O(T). Attention captures global context every 4 layers. MoE provides specialized capacity exactly where global reasoning happens.

4. Single-Line Scalability

The entire model is parameterized through one config. Scaling from 57M to 1.3B+ parameters requires changing one integer :

config = HybridModelConfig(num_layers=8)   # Titan Tiny    57.7M
config = HybridModelConfig(num_layers=16)  # Titan v1     450.4M
config = HybridModelConfig(num_layers=24)  # Titan Mid    690.0M
config = HybridModelConfig(num_layers=48)  # Titan Large  ~1.3B

The 1:3 interleaving, expert counts, RoPE — everything propagates automatically.

Training Results (Titan v1)

Config	Value
Hardware	Single NVIDIA L4 24GB
Instance	AWS g6.2xlarge
Precision	bfloat16
Effective batch	64 sequences (~65K tokens)
Peak throughput	~6,200 tok/s
Total tokens trained	~4.2B
Total cost	~$50

Validation Perplexity Progression

Step	Val Loss	PPL	Notes
12,000	2.38	10.81	50-shard val set
20,000	3.47	32.31	Expanded to 200 shards (Polish added)
26,000	3.38	29.26	Monotonic improvement resumes
42,850	3.30	~27.5	Final checkpoint

The PPL increase at step 20K is from validation set expansion (50→200 shards, adding Polish text) — not model degradation. Within the 200-shard regime loss decreases monotonically.

Titan v2 — First Training Cycle Complete

Titan v2 completed its first full pre-training cycle (3 epochs) with major improvements across every component:

Aspect	v1	v2
Parameters	450.4M	464.5M
Tokenizer	GPT-2 BPE 50K	Custom BPE 64K (PL + EN + bio + code)
Polish support	Fragmented (ą→2 tokens)	Native single-token
Optimizer	AdamW	AdamW(Tested and Stable)
Training	Single GPU	Single GPU + gradient checkpointing
Data pipeline	Runtime loading	Pre-tokenized binary shards
Data quality	Unknown	Full corruption audit — 0 binary artifacts

Titan v2 Pre-Training — Completed Cycle

Start:          Fresh init, custom 64K tokenizer
Epoch 0:        PPL  87.87  ← cold start, new vocabulary
Epoch 1:        PPL  61.37  ← −26.5 points in one epoch
Epoch 2:        PPL  57.50  ← best_val_loss = 4.0523  ✅ FINAL

Total tokens:   ~1.8B  (3 epochs × ~0.6B per epoch)
Total steps:    27,981
Hardware:       AWS g6.2xlarge (L4 24GB) + benchmarking on A100
Duration:       6 days  (Jun 11 → Jun 17 2026)
Cost:           ~$40

Four-domain corpus: 45% FineWeb-Edu (English) + 18% Polish (articles + Wolne Lektury) + 20% biomedical (PubMed/clinical) + 17% Python/code.

Currently: Building an expanded ~4.56B token dataset for the next training cycle, targeting 9B total tokens — Chinchilla-optimal scale for 464M parameters.

Looking for arXiv Endorsement

The paper is pending submission to arXiv cs.LG. As a first-time independent submitter I need one endorsement from an established arXiv author in cs.LG or cs.AI.

If you’re endorsed for cs.LG and find this work interesting — please reach out via HuggingFace messages. One endorsement is all I need.

The paper covers: architecture design, fixed-shape dispatch, EMA load balancing, training dynamics, and scaling analysis.

Happy to discuss

Why 1:3 interleaving over 1:1 or 1:7?
Fixed-shape dispatch implementation details
Training stability on a single GPU without loss spikes
The $50 compute budget — what’s possible and what isn’t
Roadmap to Titan Standard (~1B) and beyond

Built from scratch, one GPU, one researcher. Project Inkblot