Hyperparameters, scaling behavior, and regularization

Batch size, learning rate, and effective batch size

You control how fast and how noisily your small GPT learns mostly with batch size and learning rate.

• Batch size = number of tokens (or sequences) per optimizer step.
• Learning rate (LR) = how big each weight update is.
• Effective batch size = “how much data each optimizer step really sees” once you include gradient accumulation and multiple devices.

How they interact

Mechanically, one training step is:

1. Take a batch of examples.
2. Run forward pass → get loss.
3. Backprop → get gradients.
4. Update weights by something like
   $\Delta \theta \approx -\eta \cdot g$
   where $\eta$ is learning rate and $g$ is the (averaged) gradient.

Now the coupling:

• Larger batch size

  • Gradient estimate is less noisy (closer to the true gradient over the dataset).
  • You can usually use a larger learning rate safely, because the direction is more stable.
  • But each step is more expensive, and you get fewer steps per unit of compute.

• Smaller batch size

  • Gradient is noisier.
  • You need a smaller LR to avoid bouncing around or diverging.
  • You get more steps per unit of compute, each cheaper.

A common rule of thumb (not a strict law):
if you multiply batch size by $k$, you can often multiply LR by somewhere between $\sqrt{k}$ and $k$ safely, up to a point.

Effective batch size with gradient accumulation

If your GPU can’t fit a big batch, you can simulate it.

Gradient accumulation mechanism:

• Do several forward+backward passes without stepping the optimizer.
• Sum (or average) gradients.
• After N_accum mini-steps, apply one optimizer step.

If you also train on multiple GPUs, each has its own share of the global batch.

The effective batch size is:

$$\text{effective\_batch} = \text{per\_device\_batch} \times \text{num\_devices} \times \text{grad\_accum\_steps}$$

Example:

• Per-device batch = 8 sequences
• 2 GPUs
• grad_accum_steps = 4

Then:

$$\text{effective\_batch} = 8 \times 2 \times 4 = 64 \text{ sequences}$$

So your optimizer is updating weights as if you had batch size 64, even though each individual forward pass only sees 8 per GPU.

Concrete hyperparameter example (small GPT)

Assume:

• Small GPT (say, 50M–150M parameters).
• Single GPU.
• You measure batch size in tokens (typical for language models).

One reasonable starting point:

• Per-step tokens: 8k–32k tokens per optimizer step.
• If your sequences are length 512, that’s about 16–64 sequences.

Possible configs:

So for a small GPT, a good initial LR guess (Adam/AdamW):

• 1e-4 with effective batch around 1e4–1e5 tokens.
• If loss diverges (NaNs, huge spikes), halve LR.
• If training is very slow and stable, you can try 2e-4.