Words, Tokens and Embeddings

Oz Akan July 31, 2025
Source
We know LLMs don't understand words, so we need to convert words into tokens first and then tokens to embeddings. Tokens are integers and they don't map to words directly. Embeddings are N-dimensional vectors. How do one map to another? It seems confusing.

Discussion in the ATmosphere

Loading comments...