Autoregressive language modeling objective and metrics

Autoregressive factorization: from joint to conditionals

When you train a GPT-style language model, you’re really teaching it the probability of a whole token sequence by breaking it into easier conditional pieces.

We start with a sequence of tokens
$x_1, x_2, \dots, x_T$.

Autoregressive factorization writes the joint probability as:

$$p(x_1, x_2, \dots, x_T) = \prod_{t=1}^{T} p(x_t \mid x_1, x_2, \dots, x_{t-1})$$

(With the convention that $p(x_1 \mid x_{<1}) = p(x_1)$, since there’s no past.)

Why this is tractable

• The full joint $p(x_1,\dots,x_T)$ over a large vocabulary is astronomically big if you try to model it directly.
• The chain rule of probability always lets you write a joint as a product of conditionals like above.
• An autoregressive LM (e.g. GPT) only needs to output one distribution at a time:
  given a prefix $(x_1,\dots,x_{t-1})$, output $p(x_t \mid x_{<t})$ over the vocabulary.
• During training:
  • You feed in the prefix tokens.
  • The model predicts the next-token distribution.
  • You do this for each position in the sequence.
• At inference:
  • You sample $x_1$ from $p(x_1)$.
  • Then sample $x_2$ from $p(x_2\mid x_1)$.
  • And so on.

The model never has to store or enumerate the full joint distribution; it only implements a function mapping a prefix to a probability distribution over the next token. That’s what makes training and sampling tractable.

Concrete example

Say your vocabulary is tiny: ["I", "love", "cats", "."].
For the sequence: "I love cats .":

The autoregressive factorization is:

$$p(\text{"I love cats ."}) = p(\text{"I"}) \cdot p(\text{"love"} \mid \text{"I"}) \cdot p(\text{"cats"} \mid \text{"I love"}) \cdot p(\text{"."} \mid \text{"I love cats"})$$

A GPT-style model, given the input tokens:

• ["I"] → predicts $p(\cdot \mid \text{"I"})$
• ["I", "love"] → predicts $p(\cdot \mid \text{"I love"})$
• ["I", "love", "cats"] → predicts $p(\cdot \mid \text{"I love cats"})$

You can multiply the appropriate probabilities to get the sequence probability.