DAY 01
Course orientation and LM objective 2 items Learn Course structure, expectations, and project scope CS336 course goals and emphasis on systems · Assignments and how they map to a minimal GPT pipeline · Compute and dataset assumptions for self-study → Learn Autoregressive language modeling objective and metrics Autoregressive factorization p(x₁,…,x_T) into conditional probabilities · Token-level logits, softmax, and categorical distribution · Negative log-likelihood and cross-entropy loss for LMs → DAY 02
Tokenization and data pipeline basics 2 items Learn Overview of CS336 Lecture 1 and tokenization landscape Purpose of tokenization in language modeling · Character/byte-level vs subword tokenization · BPE and unigram subword tokenization concepts → Build Simple character/byte-level tokenizer and text loader Conversion of raw UTF-8 text into byte or character ids · Vocabulary construction for a byte/character tokenizer · Streaming large text files without loading entirely into memory → DAY 03
Neural network and optimization refresh 2 items Learn Softmax, cross-entropy, and gradients in language models Softmax properties and numerical stability tricks · Cross-entropy loss gradient with respect to logits · One-hot targets vs index-based loss implementations → Learn Gradient descent, overfitting, and regularization in LMs Stochastic gradient descent and mini-batching · Overfitting in language modeling · Weight decay, dropout, and early stopping → DAY 04
Transformer block conceptual design 2 items Learn Transformer architecture overview for GPT-style LMs Token embeddings and positional encodings · Self-attention as content-based addressing · Multi-head attention and parallel subspaces → Learn Causal masking and autoregressive attention Causal (triangular) attention masks · Implementation of masks in attention score computation · Difference between encoder, decoder, and GPT-style stacks → DAY 05
Implementing core attention and Transformer components 2 items Build Scaled dot-product attention implementation in PyTorch Query, key, and value linear projections · Attention score computation and scaling by √d_k · Applying causal masks before softmax → Build Multi-head attention and Transformer block skeleton Multi-head splitting and concatenation mechanics · Output projection after concatenating heads · Residual pathways around attention and MLP blocks → DAY 06
Building the full GPT-style model and training loop 2 items Build Minimal GPT-style language model class Token and positional embedding layers · Stacking multiple Transformer blocks · Final linear layer mapping to vocabulary logits → Build Basic training loop with cross-entropy loss Batching token sequences and shifting targets · PyTorch training loop structure · Gradient clipping to prevent exploding gradients → DAY 07
Systems basics: batching, data loaders, and GPU use 2 items Learn Data loading, batching, and efficient sequence packing PyTorch Dataset and DataLoader abstractions · Packing continuous token streams into batches · Shuffling and epoch definition for streaming text → Build GPU-enabled training loop with basic profiling Device placement and moving tensors to GPU · Measuring tokens-per-second throughput · Gradient accumulation to simulate larger batches → DAY 08
Scaling, regularization, and stability 2 items Learn Hyperparameters, scaling behavior, and regularization Batch size, learning rate, and effective batch size · Learning rate warmup and decay schedules · Dropout and weight decay in Transformers → Learn Debugging instability and common failure modes Detecting exploding/vanishing gradients · NaNs and Infs in activations or loss · Initialization choices for Transformer weights → DAY 09
Full pipeline training on a modest corpus 2 items Build End-to-end training on a small real-world dataset Corpus selection and preprocessing for language modeling · Integrating tokenizer, DataLoader, and model · Tracking both loss and perplexity over time → Build Basic sampling and qualitative model inspection Autoregressive sampling loop from trained GPT · Temperature scaling of logits · Top-k and top-p (nucleus) sampling basics → DAY 10
Systems-focused refinements from Assignment 2 2 items Learn CS336 systems themes: throughput and efficiency Bottlenecks in LM training pipelines · Mixed precision training concepts · Activation checkpointing and memory trade-offs → Build Micro-optimizations and instrumentation of your training loop Fine-grained timing of data loading vs model compute · Configurable precision and gradient accumulation settings · Simple experiment runner abstraction → DAY 11
Evaluation, failure modes, and limitations 2 items Learn Quantitative and qualitative evaluation of LMs Perplexity vs downstream performance · Calibration and probability quality · Error types in generated text → Learn Safety, biases, and responsible deployment basics Sources of bias in training corpora · Toxic and harmful content generation risks · Guardrails via prompting and filtering → DAY 12
Scaling experiments and ablations 2 items Build Model size and context length ablation study Effect of number of layers and heads on performance · Impact of embedding dimension and MLP size · Sequence length and its effect on context modeling → Learn Relating your experiments to scaling laws and practice Empirical scaling laws for loss vs compute · Data vs parameter vs compute-scarce regimes · Transfer from small-scale experiments to large models → DAY 13
Course synthesis and CS336 alignment check 2 items Learn Mapping your work to official CS336 content Coverage of CS336 lectures vs your implementation · Assignment 1 basics vs your core model code · Assignment 2 systems vs your optimizations → Build Consolidated, documented mini-GPT codebase Clear module structure for tokenizer, model, and training · Inline documentation of design decisions · Config-driven experiment setup → DAY 14
Final evaluation, reflection, and next steps 2 items Build Final model training run and evaluation report Selection of best-performing configuration · Comprehensive reporting of metrics and samples · Critical analysis of strengths and weaknesses → Learn Big-picture understanding of modern LMs Pretraining vs fine-tuning vs prompting paradigms · Where small from-scratch models fit in practice · Connections from CS336-scale models to frontier LLMs →