🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now

Quick clarification on the PPL numbers — I think there’s a misread in the post. The PPL ~57.5 for Titan v2 isn’t a final result — it’s a snapshot from Epoch 2 of an ongoing run. Every model starting cold with a brand new 64K vocabulary begins around PPL 100+ (Titan v2 opened at PPL 87.87 at Epoch 0, which is actually quite clean for a cold start). The post was showing the trajectory, not the destination.

Here’s where things actually stand as of today:

Cycle 1 (original dataset, ~1.8B tokens):

Epoch 0 : PPL 87.87 ← cold start, new vocabulary

Epoch 1: PPL 61.37 ← −26.5 points in one epoch

Epoch 2: PPL 57.50 ← Cycle 1 complete

Cycle 2 (expanded dataset, 3.78B tokens/epoch):

Step 45,000 : val_loss 3.8319 → PPL ~46.1 ← current BEST

Step 47,000 : autoresume checkpoint — still training

Still actively converging. Target is ~9.3B total tokens (Chinchilla optimum for 464M params).

On all your technical points — you’re right across the board, and I’m taking each one seriously:

BPB instead of PPL — absolutely, and this is now on my list before the paper submission. The v1 (50K vocab) vs v2 (64K vocab) comparison in raw PPL is misleading and I should have flagged it myself. BPB is the right cross-tokenizer metric and reviewers will ask for it.

Per-domain eval — planned for the next checkpoint report. The 45/18/20/17 mix is a hypothesis I want to validate, not a result I’ve confirmed. Per-domain BPB across checkpoints will tell me whether FineWeb is carrying everything or whether Polish and bio are genuinely learning.

The ablation — this is the one I keep circling back to. Mamba-only vs Attn+MoE-only vs triple-hybrid at matched params and tokens, run at 57M scale. It’s cheap and it’s the figure the architecture claim actually rests on. I’ll run it.

SFT as a controlled pair — same SFT data, same recipe, two bases: 1.8B checkpoint vs the 9.3B one. That’s a clean figure and I’ll run it exactly that way.

On the Chinchilla framing — noted. 9B is a checkpoint, not a ceiling. LLaMA proved empirically that for small models you train well past compute-optimal because inference is cheap and you’re buying capability, not minimizing FLOPs. If the loops persist past 9B, more tokens is the lever.

On the novelty — fair and appreciated. Jamba is right there and I cite it. The defensible claim is the fixed-shape dispatch + loss-free EMA balancing + doing it from scratch under 1B for ~$50. Leading with the engineering contribution rather than the architectural combination is the stronger move for the paper abstract.

​I’ll post a full checkpoint update this weekend—I’m taking some time off from work to focus entirely on BPB numbers and the per-domain breakdown. Would genuinely love your eyes on the Titan v2 model card if you have a moment — it has the full training config, architecture details, and I just updated it with the corrected training progress.

This is the most useful single comment this project has received.

Thank you!!!

Mati83moni