LLM Tokenization & Attention Visualizer
Watch text flow through the front of a language model. Your words are split into subword tokens, each token maps to a vector, and an attention pattern decides which tokens look at which. Adjust the context window, toggle the GPT-style causal mask, and switch heads to see the shape of the data change. The tokenizer is approximate and the vectors and attention are illustrative stand-ins, so the numbers teach the concepts rather than reproduce a real model.
Controls
Sample texts
Attention head
Context window
Fits10 of 32 tokens used. All tokens fit inside the window.
Tokens
A leading space is shown as a middle dot ("·"). Each chip is one token. Tokens past the context window are dimmed.
Token vectors
Each token maps to a vector of 16 numbers. Blue is negative, red is positive. Illustrative vectors, not learned embeddings.
Attention weights
Row i is the query token, column j is the key token. A darker cell means token i attends more to token j. Each row is a probability distribution that sums to 1. With the causal mask on, a token cannot attend to later tokens, so the upper triangle is blank.
Illustrative weights derived from token vectors, not a trained attention head.
Reference Guide
Subword tokenization
Language models do not read whole words or single letters. They read tokens, which are common chunks of text learned from a large corpus. A token can be a short word, a word fragment, a number, or a punctuation mark.
This lets a fixed vocabulary cover almost any input. Common words become a single token, while rare or made-up words break into several pieces. A leading space is usually part of the token, which is why the same word can tokenize differently at the start of a sentence.
Why tokens, not words or characters
- Words. A pure word vocabulary cannot handle new or misspelled words and would be enormous.
- Characters. Single characters keep the vocabulary tiny but force the model to process very long sequences.
- Subwords. Tokens are a middle ground that keeps sequences short while still covering any input.
- Cost and limits. Token count, not word count, drives both the price of an API call and how much fits in the context window.
Tokens as vectors
After tokenizing, each token id is looked up in an embedding table that turns it into a vector of numbers. Similar tokens end up with similar vectors, so the geometry of the vector space carries meaning.
The vectors shown here are illustrative. They are generated deterministically from the token id so you can see the shape of the data, but they are not trained embeddings and carry no real semantics.
Self-attention and query, key, value
Attention lets each token gather information from other tokens. Every token produces a query, a key, and a value. The query of one token is compared against the keys of all tokens to score how relevant each one is.
Those scores are turned into weights with a softmax, so each row of the attention matrix is a probability distribution that sums to 1. The output for a token is a weighted blend of the value vectors it attended to.
Causal masking
A GPT-style model predicts the next token, so a token must not peek at tokens that come after it. The causal mask blocks every future position before the softmax, which leaves the attention matrix lower triangular.
Turn the mask off in the tool and the upper triangle fills in, showing a bidirectional pattern more like an encoder model. Turn it on and each token only attends to itself and the tokens before it.
The context window
A model can only attend over a fixed number of tokens at once. That budget is the context window. When the input is longer, the oldest tokens fall outside the window and the model can no longer use them.
Lower the window size in the tool and watch tokens past the limit dim out. This is why long documents are summarized or chunked before they are fed to a model.
What this tool does and does not do
This is a teaching illustration, not a running model. The tokenizer approximates a subword scheme using a small built-in vocabulary, so its splits and token ids are illustrative and will not match OpenAI or any other production tokenizer.
The vectors and attention weights are computed deterministically from the tokens with a fixed pseudo-random rule. They reproduce the correct shape of the data flow, text to tokens to vectors to attention, and the correct properties, such as each attention row summing to 1 and the causal mask zeroing future positions. They do not reflect learned meaning. For exact token counts, use a tokenizer built for your specific model.
