End-to-end training on a small real-world dataset · CS336 Language Modeling From Scratch

Deliverableartifact

A main.py (or notebook) that runs the full pipeline—tokenization, data loading, training, validation—on a chosen text corpus and outputs logged metrics and saved checkpoints.

Goals

Train your GPT model on a small corpus (e.g., project Gutenberg text or similar) until validation loss clearly decreases.
Plot or print training and validation loss/perplexity over time and interpret trends.
Confirm that your pipeline can be re-run deterministically given a fixed seed and config.
Document your experiment configuration (model size, LR, batch size, dataset) for future reference.

Resources

📄stanford-cs336/assignment1-basics 📄bcefghj/learn-cs336

Overview: what you’re building and why

You’re going to build a single script (or notebook) that does a full language-model training run on a small real dataset:

picks and cleans a text corpus
tokenizes it
builds PyTorch Dataset/DataLoader
defines and trains a tiny language model
tracks loss + perplexity
saves checkpoints + logs + config for reproducibility

Assumptions:

You’re using Python and PyTorch (plus torchvision-style patterns, but for text).
You’re okay with CPU or a single GPU; we’ll keep the model tiny.

All code examples are complete enough that you can paste them into main.py and run (after installing dependencies).

1 / 6