🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now

So sorry if answer will be to long but honestly was to many surprises on this project and dont know where to start answer hehe Let’s do from question about interacting with it — yes, quite a lot actually! Just ran a fresh inference test tonight across all four domains the model was trained on. Here’s the honest picture: What works already (after just 1.8B tokens, no SFT): Polish diacritics render perfectly — ą ę ć ł ń ó ś ż ź all clean, including Polish quotation marks „ ". This was the whole point of building a custom 64K tokenizer — GPT-2 fragmented Polish characters, Titan v2 doesn’t. Domain awareness is there — medical prompts produce real terminology (RNA-seq, differential gene expression, chromosome references, Fig. 2 citation format). It knows it’s in a scientific paper. Python syntax surface is correct — def, docstrings, imports, for loops — the structure is right even if the logic isn’t. What doesn’t work yet — and this is expected: Semantic loops (“future of the future of the future”) — classic undertrained behaviour, fixable partially with repetition_penalty without retraining. No real algorithmic reasoning in code — Fibonacci with no Fibonacci logic, factorial accepting a string. Pattern matching, not understanding. Polish text has an archaic 19th-century flavour with was bit funny— that’s the Wolne Lektury (classic Polish literature) dataset leaving its fingerprint. What’s next: First SFT attempt already happened on the 1.8B checkpoint — curious results, data collected. But the real milestone is tomorrow: Titan v2 hits Chinchilla optimum (~9.3B total tokens for 464M parameters) completing epochs 3 and 4 on the expanded dataset. That’s the point where the model should have seen enough data to actually justify its parameter count. After that — a proper SFT run on a Chinchilla-trained base. The hypothesis is that instruction tuning on top of an undertrained model mostly amplifies existing loops, while SFT on a properly trained base should produce something meaningfully different. Tomorrow we find out if that hypothesis holds. Either way, it’s data. That’s the fun part of doing this from scratch — every stage teaches you something the papers don’t mention

BEST REGARDS

Mati83moni