Simple character/byte-level tokenizer and text loader · CS336 Language Modeling From Scratch

Deliverableartifact

A Python module (e.g., tokenizer.py) that defines a byte-or-char-level tokenizer plus a data-loading function that emits fixed-length input/target sequences from a text file.

Goals

Show that your tokenizer can encode and decode arbitrary ASCII and some UTF-8 text losslessly.
Demonstrate a function that yields (input, target) sequences for LM training from a continuous token stream.
Explain how your loader would handle a corpus too large for RAM.
Print example token ids and mapped-back text to verify correctness.

Resources

📄stanford-cs336/assignment1-basics 📄bcefghj/learn-cs336

Overview: what you’re building

You’ll build a single Python module (say tokenizer.py) that can:

Turn raw UTF-8 text into byte or character IDs and back (deterministic, invertible).
Define a vocabulary with a known ID range and a couple of special tokens.
Stream a big text file chunk by chunk (no full-file-in-RAM).
Pack the token stream into fixed-length (input, target) sequences ready for autoregressive training.

The end result will be something you can drop into a tiny language-model training loop later.

1 / 6