What Is a Language Model?
A language model is a system designed to work with language by learning patterns in text.
At a basic level, a language model learns relationships between pieces of language and uses those learned patterns to make predictions.
Modern language models can perform tasks such as answering questions, summarizing text, translating languages, writing content, and generating computer code.
But how can a model generate language that appears so natural?
One of the most important ideas to understand is next-token prediction.
Predicting What Comes Next
Imagine that you see the beginning of a sentence:
"The sun rises in the"
You can probably predict that the next word is likely to be "east".
A language model performs a much more sophisticated version of this idea.
It analyzes the input it has received and calculates probabilities for possible next tokens.
It then selects or samples a token and continues generating.
What Is a Token?
Language models generally do not process text exactly as humans see it.
Text is converted into smaller units called tokens.
A token might represent a whole word, part of a word, punctuation, or another piece of text.
The exact way text is divided depends on the tokenizer used by the model.
For example, a common word might be represented by one token, while a less common or complex word might be divided into several tokens.
This allows language models to work with a very large vocabulary while keeping the number of individual units manageable.
From Text to Numbers
Neural networks operate on numerical representations rather than raw text.
When you enter a sentence into a language model, the text is first tokenized.
The tokens are then converted into numerical representations that the neural network can process.
This allows the model to perform mathematical operations on the information contained in the text.
What Does the Model Learn?
During training, a language model processes enormous amounts of text and learns statistical patterns in language.
It learns relationships between words and other tokens, grammatical structures, common phrases, concepts, and many other patterns present in the training data.
The model does not simply store a giant list of answers.
Instead, training adjusts a very large collection of numerical parameters so that the model becomes better at predicting what is likely to come next.
Parameters
Parameters are learned numerical values inside a machine learning model.
They are adjusted during training and influence how the model processes information and produces predictions.
Large language models can contain very large numbers of parameters.
It is tempting to think of parameters as individual pieces of knowledge, but that is not an accurate mental model.
Knowledge and patterns are represented across many parameters and their interactions.
What Happens When You Ask a Question?
Suppose you type:
"What is artificial intelligence?"
A simplified version of what happens is:
- Your text is received by the application.
- The text is converted into tokens.
- The tokens are converted into numerical representations.
- The neural network processes the information.
- The model calculates probabilities for possible next tokens.
- A token is selected.
- The new token becomes part of the growing response.
- The process repeats until the response is complete.
Generating One Token at a Time
One useful mental model is that a language model generates a response incrementally.
Imagine the model begins with:
"Artificial intelligence is"
The model predicts a likely next token.
Perhaps the next token contributes the word "the".
Now the sequence contains more information.
The model predicts another token.
This process continues repeatedly.
The final response is the result of many sequential predictions.
Does the Model Choose Only the Most Likely Word?
Not necessarily.
A model can assign probabilities to many possible next tokens.
The system can use different generation strategies to select the next token.
In some situations, choosing a highly probable token may be appropriate.
In other situations, allowing some variation can produce more diverse responses.
The exact behaviour depends on the model and the generation settings used by the application.
Why Does Context Matter?
Consider these two questions:
"What is Python?"
and:
"What is Python used for in web development?"
The second question contains additional context that helps determine the intended meaning.
Language models use the surrounding tokens to help determine what the input means and what response is appropriate.
Context is therefore extremely important.
Conversation Context
When you have a conversation with an AI assistant, earlier messages can provide context for later messages.
For example:
User: "I am learning Python."
User: "What should I learn next?"
The second question can be interpreted differently because the previous message provides context.
In an AI application, conversation history may be included in the information sent to the model.
Context Has Limits
Language models cannot necessarily process an unlimited amount of conversation at once.
A model has a finite context window, which represents how much information can be processed together for a particular interaction.
Modern models can support very large context windows, but there are still practical limits.
This is one reason long conversations or very large documents may need special handling.
Training vs Generating a Response
It is important to separate training from ordinary response generation.
During training, the model learns patterns from large amounts of data and its parameters are adjusted.
During inference, the trained model processes new input and generates an output using the parameters it already has.
When you ask a normal question, the model is generally performing inference rather than retraining itself from scratch.
Does a Language Model Store Every Sentence?
It is tempting to imagine that a language model simply stores every sentence from its training data and retrieves the correct one when asked a question.
That is not a good general description of how these models work.
The training process adjusts parameters so that patterns in the training data become represented in the model.
The model can then use those learned patterns to generate new sequences of tokens.
This is one reason a language model can produce combinations of words that were not necessarily present as exact sentences in its training data.
Why Can It Generate New Text?
Because the model has learned patterns and relationships, it can combine them in new ways during generation.
For example, it may have learned patterns associated with explanations, programming, stories, questions, instructions, and many other forms of language.
When given a new request, the model uses its learned parameters and the current context to generate a sequence that fits the request.
Language Models Are Prediction Systems
One of the most useful ideas to remember is that language models are fundamentally prediction systems.
They predict what token is likely to come next based on the tokens and context available to them.
This simple description hides enormous technical complexity.
Modern language models contain sophisticated neural network architectures and are trained on enormous datasets.
But the idea of predicting the next token provides a useful foundation for understanding them.
Why Prediction Can Produce Useful Answers
You might wonder how predicting the next token can result in an answer to a complicated question.
The reason is that the model has learned many layers of relationships within language and other information during training.
When a question is provided, the model uses the context to predict a sequence of tokens that forms a response.
When this process is repeated across many tokens, the result can be a paragraph, explanation, story, program, or other form of content.
What About Facts?
A language model can produce factual information because patterns related to information can be represented in its learned parameters.
However, this does not mean that every generated statement is guaranteed to be correct.
A model can produce incorrect information while generating text that sounds convincing.
Understanding this limitation will become important later in this module.
Large Language Models
The word large in large language model generally refers to the scale of the model and its training.
Large language models can contain very large numbers of parameters and can be trained using enormous collections of data and substantial computing resources.
The increased scale can allow models to learn complex patterns and perform many different language tasks.
From Language Models to AI Assistants
A language model by itself is not necessarily the same thing as a complete AI assistant application.
An AI assistant can include many components around the model.
- User interface
- Conversation management
- Safety systems
- Tools
- Search or retrieval
- Memory or stored information
- The underlying language model
The application can combine these components to provide a useful experience.
The Big Picture
We can now connect the ideas from the previous lessons:
Data → Training → Neural Network → Learned Parameters → Tokens + Context → Next-Token Prediction → Generated Output
This is a simplified picture, but it gives us a useful foundation for understanding modern language-based AI.
What You Should Remember
- Language models learn patterns in language from training data.
- Text is divided into tokens that the model can process.
- Neural networks operate on numerical representations of those tokens.
- Training adjusts model parameters.
- During inference, the model uses its learned parameters to process new input.
- Next-token prediction is a fundamental part of language generation.
- Context helps determine the meaning and appropriate continuation of text.
- Language models can generate useful text but can still produce incorrect information.
What Comes Next?
We now have a basic understanding of how language models process tokens and generate text.
But one major question remains:
How does a modern language model know which parts of a long piece of text are important when predicting the next token?
To understand that, we need to explore tokens, context, and attention.