Start with a straight line
In y = mx + b, the slope m is the number you adjust until the line fits the data. In a neural network the same quantity is called a weight. Training is the process of adjusting billions of them at once.
One neuron is a weighted vote
An artificial neuron multiplies each input by a weight, adds the results together, adds an offset (the b in the line), and passes the total through a simple nonlinear function.
That last step is what lets layers add capability. Without it, any stack of layers collapses back into a single straight line, however many layers there are.
Billions of them
A model is billions of those weights, in layers. They are also called parameters: the numbers in a model that determine how an input becomes an output.
The weights are the only thing training produces. Everything a model learned is in those numbers, and no one can read a reason out of any single one. That is why a learned system cannot have its logic audited the way a rule system can, and why publishing a model's weights reveals so little about what went into them.
What a transformer is
The transformer is the architecture behind current language models. Google researchers published it in 2017, and most of the improvement in generative systems since then rests on it.
A transformer processes whole sequences at once. It uses a technique called attention to weigh how much each part of an input bears on every other part. For almost every purpose inside an organization, that is as deep as the architecture needs to go.
Where this shows up in practice
- Explainability. Asking why a learned model decided something is asking someone to read its weights. Ask for test results and a record of behavior, which can be produced.
- Size. A parameter count is a count of weights. It says how much there is to store and move, and it no longer predicts running cost. See Does all AI need a hyperscale data center?
- Control. Whoever holds the weights holds the model. An open-weight release hands them over; a hosted model keeps them with its provider.
Frequently asked questions
- What is a weight in an AI model?
- A weight is a number that survived training. In the line y = mx + b, the slope m is the number you adjust until the line fits the data, and in a neural network the same quantity is called a weight. Weights are also called parameters, and a model is billions of them arranged in layers.
- Why can't anyone read what a weight means?
- The weights are the only thing training produces, and everything a model learned is in those numbers. No one can read a reason out of any single one, including the people who trained the model. That is why a learned system cannot have its logic audited the way a rule system can.
- What is a transformer?
- The transformer is the architecture behind current language models. Google researchers published it in 2017, and most of the improvement in generative systems since then rests on it. It processes whole sequences at once and uses a technique called attention to weigh how much each part of an input bears on every other part.
- Does a parameter count tell you what a model costs to run?
- Not any more. A parameter count is a count of weights, so it says how much there is to store and move, and it no longer predicts running cost. Techniques such as lower precision, adapters, and sparse activation have broken the link between a model's size and what it takes to run.
- What does it mean to hold a model's weights?
- Whoever holds the weights holds the model. An open-weight release hands them over, so you can download, run, and host the model yourself, while a hosted model keeps them with its provider. Weights can be copied, measured, and adjusted, and the model they make can be tested, but nobody can read them.