Tokenization & Embeddings

The Fundamental Problem¶

Neural networks only understand numbers.

Not words. Not characters. Not meaning. Just floating-point numbers arranged in vectors and matrices.

So before a transformer can do anything with text, we need to convert it into numbers. But not just any numbers—we need representations that capture something useful about what words mean and how they relate to each other.

This notebook covers that conversion: from the string “I like transformers” to a matrix of numbers the model can actually process.

Step 1: Tokenization (Text → Integer IDs)¶

The first step is simple: break text into pieces and assign each piece a number.

These pieces are called tokens. In real models, tokens might be words, parts of words (“un” + “believe” + “able”), or even individual characters. GPT-3 uses about 50,000 different tokens. We’ll use 6.

Our vocabulary:

The special tokens (<PAD>, <BOS>, <EOS>) are conventions that help the model understand sequence boundaries. <BOS> says “a new sequence starts here.” <EOS> says “the sequence ends here.” <PAD> fills in gaps when sequences have different lengths.

Tokenizing Our Input¶

Let’s convert “I like transformers” into token IDs:

Original text:   "I like transformers"
With markers:    <BOS> I like transformers <EOS>
Token IDs:       [1,   3, 4,   5,           2]

We add <BOS> at the start and <EOS> at the end. The result is a list of 5 integers.

The Language Modeling Task¶

Before we go further, let’s be clear about what the model is trying to do.

A language model predicts the next token. At each position, it looks at all the previous tokens and guesses what comes next. This is called autoregressive generation—each prediction depends only on what came before.

For our sequence:

Position 4 is <EOS>, which marks the end—nothing to predict.

This is why we need a causal mask later (we’ll see it in the attention notebook). The model at position 2 shouldn’t be able to peek at position 3 and 4—that would be cheating.

Step 2: Token Embeddings (IDs → Vectors)¶

Okay, we have integers. But integers are terrible representations for learning.

Why? Because integers imply ordering and distance that doesn’t exist. Token 5 isn’t “bigger” than token 3. Token 4 isn’t “between” tokens 3 and 5 in any meaningful way. The numbering is arbitrary.

What we need is a continuous vector for each token—something we can do math with, something where distance and direction have meaning.

Enter: embeddings.

The Embedding Matrix¶

We create a matrix $E_{token}$ of shape [vocab_size, d_model] = [6, 16]. Each row is a 16-dimensional vector representing one token:

To get the embedding for token ID $i$, we just look up row $i$. That’s it. Embeddings are just table lookups.

Why Random Initialization?¶

These embeddings start as random noise. They don’t encode any meaning yet.

That’s fine. During training, the model will adjust these vectors so that:

• Similar words end up with similar embeddings

• The vectors capture useful features for the prediction task

• Relationships between words are encoded in the geometry (e.g., king - man + woman ≈ queen)

We initialize with small random values (scale=0.1) to break symmetry and avoid numerical issues at the start of training.

Step 3: Position Embeddings (Where in the Sequence?)¶

Here’s something weird about transformers: they have no built-in sense of order.

Think about it. The attention mechanism (which we’ll see later) compares every token to every other token simultaneously. It’s not processing left-to-right like a human reading. It’s looking at all tokens at once.

This means: without something extra, the model has no idea that “I” comes before “like” which comes before “transformers.” It would process “transformers like I” exactly the same way.

That’s... bad. Word order matters.

The Solution: Add Position Information¶

We give each position its own embedding vector, then add it to the token embedding:

The position embedding matrix $E_{pos}$ has shape [max_seq_len, d_model] = [5, 16]. Each row is a unique vector for that position.

After adding position embeddings:

• The word “I” at position 1 has a different representation than “I” at position 3

• The model can learn that certain patterns occur at certain positions

Why Add Instead of Concatenate?¶

You might wonder: why add the position embedding to the token embedding? Why not concatenate them (stick them side by side)?

Concatenation would work, but it has a cost: it would increase the dimension. If token embeddings are 16-dimensional and position embeddings are 16-dimensional, concatenation gives you 32-dimensional vectors. Every subsequent layer would need to be larger.

Addition is cheaper. We keep the same dimension, and the model learns to “share” the 16 dimensions between token identity and position information.

(There’s also a deeper reason: addition lets the model learn to ignore position when it doesn’t matter, by having position embeddings that are nearly orthogonal to the directions the model cares about. But that’s getting into the weeds.)

Putting It Together: The Input Matrix X¶

Now we can build the actual input to our transformer.

For each position $i$ in our sequence:

1. Look up the token embedding: $E_{token}[\text{tokens}[i]]$

2. Look up the position embedding: $E_{pos}[i]$

3. Add them together: $X[i] = E_{token}[\text{tokens}[i]] + E_{pos}[i]$

Let’s compute this for our sequence [1, 3, 4, 5, 2]:

The Resulting Matrix¶

We now have our input matrix $X$ with shape [seq_len, d_model] = [5, 16].

Each row is a 16-dimensional vector representing one token at one position. This matrix is the input to the transformer block.

What We’ve Built¶

Let’s recap the transformation:

"I like transformers"
        ↓ tokenization
[1, 3, 4, 5, 2]                    # 5 integers
        ↓ token embeddings
[5, 16] matrix                     # 5 rows × 16 columns
        ↓ add position embeddings
[5, 16] matrix (X)                 # final input

We’ve gone from a string to a matrix of continuous values. The model can now:

• Do math with these vectors (add, multiply, take dot products)

• Learn to adjust the embeddings during training

• Use the position information to understand word order

This matrix $X$ will flow through the rest of the transformer: attention, feed-forward layers, layer normalization, and finally produce predictions.

What’s Next¶

The input matrix $X$ is ready. Now comes the core of the transformer: self-attention.

Before we can compute attention, we need to project $X$ into three different representations:

• Query (Q): “What am I looking for?”

• Key (K): “What do I contain?”

• Value (V): “What information should I pass along?”

These projections are called QKV projections, and they’re the subject of the next notebook.