Why tokenization matters in CS336
In CS336 you’re building language models “from scratch,” and tokenization is the first hard design choice between raw text and what the model actually sees as numbers. Lecture 1 of CS336 and its notes (for example these course notes and the Lecture 1 video) emphasize that if you don’t understand tokenization, you don’t really understand what your model is modeling.
At a high level:
- Tokenization turns text (
"hello world") into a sequence of discrete tokens ([15496, 995]) that a transformer can embed and process. - How you tokenize decides:
- How long your sequences are (in tokens, not characters).
- Which patterns are “atoms” the model can learn directly vs having to compose.
- What happens when the text is weird: emojis, code, other languages, typos.
1 / 9