Causal masking and autoregressive attention

Causal (triangular) attention masks

Causal masking is what makes an autoregressive language model only use the past and present, never the future, when predicting a token.

What a causal mask looks like (length 4)

For a sequence of length 4, we index positions as 1, 2, 3, 4.
Token at position i is only allowed to attend to positions 1..i.

A causal mask matrix M of shape 4 × 4 can be written as:

• 0 = allowed (keep the attention score)
• -∞ (or a very large negative number) = blocked (zero probability after softmax)

One common convention is:

$$M = \begin{bmatrix} 0 & -\infty & -\infty & -\infty \\ 0 & 0 & -\infty & -\infty \\ 0 & 0 & 0 & -\infty \\ 0 & 0 & 0 & 0 \end{bmatrix}$$

Row = query position (token that is “looking”).
Column = key position (token being looked at).

Interpreting row 3 (i = 3):

• It can attend to positions 1, 2, 3 → 0, 0, 0
• It must not attend to position 4 → -∞

In code (PyTorch-style):

pythonimport torch n = 4 # mask[i, j] = 0 if j <= i else -inf mask = torch.zeros(n, n) mask = torch.triu(torch.full((n, n), float("-inf")), diagonal=1) print(mask)

This prints:

texttensor([[0., -inf, -inf, -inf], [0., 0., -inf, -inf], [0., 0., 0., -inf], [0., 0., 0., 0.]])

Why this shape matters

• The mask is lower-triangular (including the diagonal).
• Every row enforces: “I can only see myself and things before me.”
• That’s what keeps generation left-to-right and causal.