Words, Tokens and Embeddings
Oz Akan
July 31, 2025
We know LLMs don't understand words, so we need to convert words into tokens first and then tokens to embeddings. Tokens are integers and they don't map to words directly. Embeddings are N-dimensional vectors. How do one map to another? It seems confusing.
Discussion in the ATmosphere