Relating your experiments to scaling laws and practice

Empirical scaling laws for loss vs compute

You want to use your experiments to make smarter scaling decisions, not just collect charts. Scaling laws give you a mental model for what will probably happen before you spend more compute.

At a high level, for language models:

• As you increase compute (more parameters, more data, more steps),
  the loss tends to fall in a smooth, predictable way — roughly like a straight line on a log-log plot.
• This continues until you run into a limit:
  • not enough data (data-scarce),
  • not enough parameters (parameter-scarce),
  • or not enough total training compute (compute-scarce).

The basic shape: loss vs compute

Imagine you plot validation loss vs total training compute on log scales:

• Between A and C, loss tends to decrease smoothly as you add compute.
• Around D, the curve flattens:
  • extra compute gives diminishing returns because some other resource is now the bottleneck.

You can think of (very roughly):

$$L(C) \approx a \cdot C^{-b} + L_{\text{irreducible}}$$

• $L(C)$: loss at compute $C$
• $a, b$: constants for your setup
• $L_{\text{irreducible}}$: the floor set by data noise / task difficulty

You don’t need exact equations; what matters:

• Smoothness: if you double compute, loss changes in a pretty regular way.
• Diminishing returns: each extra unit of compute gives less improvement than the previous.

A concrete, reproducible example

Say you train the same architecture with different total tokens processed:

• Model: 40M parameters transformer
• Dataset: shuffled corpus, effectively infinite stream
• Batch tokens: 65,536

You run:

If you plot log(compute) vs loss, points A–D will sit close to a straight line.
That line is your empirical scaling law for this particular setup.

Where it breaks

Scaling regularities break or change when you:

• Change the regime: e.g., you start reusing the same data too many times (data bottleneck).
• Change architecture or training recipe a lot: different optimizer, normalization, etc. can shift the curve.
• Hit hardware or batch-size limits: if you’re forced into tiny batches with lots of noise, trends may get messy.