Transformer architecture overview for GPT-style LMs

Token embeddings and positional encodings

You’re working with sequences of token IDs (integers). A Transformer can’t use raw IDs, so the first job is: turn them into vectors and inject order.

What this is / why it matters

Transformers are permutation-invariant by default: without positions, they see a bag of tokens, not a sequence. Embeddings give each token meaning; positional encodings tell the model where it is.

From token IDs to embeddings

Assume:

• Sequence length: T
• Vocabulary size: V
• Embedding dimension (a.k.a. model dimension): d_model

You start with token IDs:

• Input IDs shape: [T]
  Example: x = [12, 57, 389, 5] (so T = 4)

You have an embedding matrix:

• E_token shape: [V, d_model]

You look up each token ID in E_token:

• Resulting token embeddings: X_token shape [T, d_model]

Concretely, if d_model = 8 and T = 3, you might get:

textX_token = [ [ 0.1, -0.3, 0.4, 0.0, 0.2, -0.1, 0.6, 0.3 ], // token 1 [-0.2, 0.5, 0.1, 0.7,-0.4, 0.2, 0.0,-0.1 ], // token 2 [ 0.0, 0.1, -0.5, 0.3, 0.4, 0.9,-0.3, 0.2 ], // token 3 ]

Why positional encodings are necessary

Self-attention computes similarity between token vectors. If you shuffle tokens, you just reorder the rows in X_token. Attention itself has no built-in sense of “this came first”.

So you add positional encodings:

• E_pos shape: [T_max, d_model] (or more, if you support longer sequences)
• For each position t (0..T-1), you get a vector E_pos[t] and add it:

$$X = X_{\text{token}} + E_{\text{pos}}$$

• Final input to first layer: X shape [T, d_model]

Mechanism:

• Each position has its own vector direction in d_model space.
• Adding it shifts the token embedding so “word at position 3” is different from “same word at position 10”.
• Attention weights now implicitly depend on position: similarity compares (token + position) pairs.

Where this breaks:

• If you didn’t add any positional info, the model couldn’t reliably distinguish “dog bites man” from “man bites dog” based on order.
• If your positional encodings don’t cover long enough sequences, positions beyond that length all look the same (or undefined), so performance collapses there.

A tiny runnable-style example (shapes)

Let:

• T = 4
• V = 50_000
• d_model = 16

Then:

• Token IDs: [4]
• E_token: [50_000, 16]
• After embedding lookup: X_token: [4, 16]
• Positional encodings: E_pos: [4, 16]
• Final input to Transformer layer: X = X_token + E_pos: [4, 16]

This is the common starting point for a GPT-style stack.

Visual: input with positions