Why Do Tokens Matter?
In the previous lesson, we learned that language models do not process text exactly as humans see it.
Instead, text is divided into smaller units called tokens.
A token may represent a complete word, part of a word, punctuation, or another piece of text.
Tokenization gives the neural network a practical way to convert language into numerical information that it can process.
A Simple Token Example
Consider the sentence:
"AI is changing the world."
A tokenizer might divide this into several tokens representing words and punctuation.
The exact tokenization depends on the tokenizer used by the model.
It is therefore better to think of a token as a unit used by a particular language-processing system rather than assuming that every token is exactly one word.
From Tokens to Numbers
Neural networks work with numerical values.
After text has been tokenized, the tokens are converted into numerical representations.
These numerical representations can then be processed by the neural network.
This transformation is one of the first steps that allows a language model to work with text.
What Is Context?
Context is the surrounding information that helps determine the meaning of the current input.
Consider the sentence:
"The bank was beside the river."
Here, the surrounding words help us understand that "bank" refers to the land beside a river rather than a financial institution.
Language models also rely heavily on surrounding tokens when processing language.
Context Changes Meaning
Consider the word "apple".
In the sentence:
"She ate an apple after lunch."
the word refers to a fruit.
In a sentence about technology, such as:
"Apple released a new computer."
the surrounding context points toward the technology company.
The same word can therefore have different meanings depending on its context.
What Is a Context Window?
A language model has a limited amount of information that it can process together for a particular interaction.
This available amount of information is commonly described as the model context window.
The context can include the current user message, previous conversation messages, instructions, and other information supplied to the model.
Modern models can support very large context windows, but the context is still finite.
Why Does Context Matter?
Imagine asking:
"What does it mean?"
Without knowing what "it" refers to, the question is incomplete.
If the previous conversation was about machine learning, the meaning may become clear.
Context allows a language model to use surrounding information when interpreting and generating text.
The Challenge of Long Text
Imagine a document containing thousands of sentences.
If the model is asked to answer a question about that document, different parts of the document may have different levels of relevance.
The model needs a way to process relationships between tokens efficiently.
This is where the idea of attention becomes important.
What Is Attention?
In modern neural networks for language, attention is a mechanism that allows the model to consider relationships between different parts of the input.
Instead of treating every token as equally important for every calculation, attention allows the model to assign different levels of importance to different tokens.
This helps the network determine which pieces of information are more relevant to the current processing step.
A Simple Example of Attention
Consider:
"The dog chased the ball because it was moving."
To interpret the word "it", we need to consider other words in the sentence.
The model can use relationships between tokens to help determine which earlier words are relevant.
Attention provides a mechanism for representing these relationships.
Attention Is Not Human Attention
The word "attention" can be misleading.
In an AI model, attention is a mathematical mechanism.
It does not mean that the model is consciously focusing on something in the same way a human does.
It means that the model calculates relationships and assigns different numerical importance to different pieces of information.
Why Attention Was Important
Earlier approaches to language processing often had difficulty handling relationships across long sequences.
Attention made it much easier for neural network architectures to consider relationships between different positions in a sequence.
This became one of the key ideas behind the Transformer architecture.
Transformers
A Transformer is a neural network architecture that makes extensive use of attention mechanisms.
Transformers became extremely important in modern AI because they can process relationships between tokens efficiently and can be scaled to very large models.
Many modern language models are based on Transformer architectures or architectures derived from the same fundamental ideas.
Self-Attention
One particularly important concept is self-attention.
Self-attention allows tokens within the same sequence to interact with one another.
For example, when processing a sentence, a token can use information from other tokens in that sentence to help determine its representation.
This allows the network to model relationships between different parts of the input.
Why Relationships Matter
Language contains many relationships that cannot be understood by looking at a single word in isolation.
Pronouns can refer to earlier nouns.
Words can modify other words.
A sentence can contain conditions, comparisons, causes, and consequences.
Attention helps neural networks represent these relationships.
Attention and Long-Range Relationships
Consider a paragraph where an important subject is introduced near the beginning and discussed again several sentences later.
A language model may need to connect those distant pieces of information.
Attention provides a mechanism for considering relationships between tokens that are far apart within the available context.
Attention Does Not Mean Perfect Understanding
Although attention is powerful, it does not guarantee that a model will understand every relationship correctly.
A model can still misinterpret a sentence, ignore important information, or produce an incorrect answer.
Attention is a mechanism that helps the network process information. It is not a guarantee of reasoning or factual accuracy.
Context and Attention Work Together
The context window determines the information that is available to the model.
Attention helps the model determine relationships between the tokens within that available information.
A simplified way to think about this is:
Context provides available information → Attention helps connect relevant information → Neural network produces useful representations and predictions.
Why Token Limits Matter
Because context windows are finite, very long inputs may require special handling.
An application might divide a large document into sections, retrieve relevant passages, or summarize earlier information.
These techniques allow AI applications to work with information that is larger than a single model context window.
Attention in Modern AI
Attention is one of the central ideas behind modern language models.
It helps neural networks process relationships between tokens and is a fundamental component of Transformer-based systems.
Understanding attention makes it easier to understand why modern language models can handle long and complicated pieces of text.
A Useful Mental Model
Imagine that every token in a sentence can look at other tokens and ask, in a mathematical sense:
"Which other pieces of information are useful for understanding my current context?"
The attention mechanism calculates relationships that help answer this question.
Again, this is an analogy. The model is not consciously asking a question.
From Attention to Modern AI Assistants
When you ask a modern AI assistant a question, the system may process many tokens representing your current request and relevant conversation context.
Transformer-based neural networks use attention mechanisms to process relationships between those tokens.
The model then generates an output based on its learned parameters and the available context.
The Big Picture
We can now connect the ideas from this module:
Data → Training → Neural Network → Tokens → Context → Attention → Prediction → Output
This simplified chain provides a useful conceptual foundation for understanding modern language AI.
What You Should Remember
- Tokens are units of text processed by language models.
- Tokens are converted into numerical representations for neural network processing.
- Context provides surrounding information that helps determine meaning.
- A context window represents the amount of information available to the model for an interaction.
- Attention is a mathematical mechanism for representing relationships between different tokens.
- Self-attention allows tokens within a sequence to interact with one another.
- Transformers rely heavily on attention mechanisms.
- Attention improves the models ability to process relationships, but it does not guarantee perfect understanding or accuracy.
What Comes Next?
We now understand how tokens, context, and attention fit together.
But there is another important distinction we need to understand:
What exactly happens when an AI model is being trained, and what happens when it is simply answering a user?
That is the subject of our next lesson: Training vs Inference.