The Broken Token: Tokenization for Malayalam Language Models

Standard LLMs fragment Malayalam words into 15+ meaningless pieces, destroying the semantic signal required for learning. This post details the training of custom BPE and Unigram tokenizers, and explores why resolving fragmentation is the necessary first step toward solving the larger problems of data scarcity and complex morphology