The Basis of Language Models: Probability

Start at the top left. English letters plus the space give 27 symbols; counting them gives letter frequencies, and dividing by the total gives letter probabilities. The black arrow leads to words: word probabilities on their own make a unigram model, and in the blue note beside it, word frequencies are also a way to tell AI-written text from human writing. In the middle, conditional probability adds context: what comes next given what came before. The arrow down leads to generating sentences from word probability tables. The note at the bottom right is my main takeaway.

The simplest language model works on characters. Take the 26 English letters plus the space, 27 symbols in all, and count how often each one appears in a text. That count is the frequency. Divide it by the total number of characters and you get the letter's probability:

P(letter) = count of that letter ÷ total number of characters

Two properties hold for any text. Each probability lies between 0 and 1, and the 27 probabilities add up to exactly 1, because every character has to be one of the 27 symbols. In ordinary English the space is the most frequent symbol, followed by vowels such as e and a, while letters like z, q, and x are rare.

✎ reading noteReading note: P(U | Q) is the probability of U given that the previous letter was Q, so the transition runs Q → U; the small diagram draws the arrow the other way. Some letters are also missing from the matrix labels; the real matrix has one row and one column for each of the 27 symbols. With the counts shown, P(space) = 14 ÷ 103 ≈ 0.136.

The same counting works one level up. Treat each word as a unit, count how often it appears in a large collection of text (a corpus), and divide by the total number of words. Two terms help here:

A model that uses only these word probabilities is a unigram language model. It looks at word frequency alone and ignores order, so it would score "the cat sat" and "sat the cat" as equally likely. To fix that, a model needs context.

Sentences are also marked with two special symbols: <s> at the start and </s> at the end. They let the model learn which words tend to begin a sentence and when a sentence is likely to stop. SLP3, Chapter 3

✎ reading noteReading note: in Jurafsky and Martin's textbook, the Berkeley Restaurant Project sample has 9,332 sentences and a vocabulary of 1,446 word types, not "1,600+ word types and 10,000+ sentences." The 33% for "I want" is P(want | I) = 0.33 from that corpus.

A conditional probability asks how likely something is once we know what came before. It is written with a vertical bar, read "given":

• P(a | e): the probability that the next letter is a, given that the current letter is e.
• P(red | big): the probability that the next word is red, given that the current word is big.

For letters, the 27 × 27 combinations form a table. Some cells are almost certain: after q, English nearly always continues with u. For words, a model that conditions on just the previous word is a bigram model, and its probabilities also come from counting:

P(red | big) = count of "big red" ÷ count of "big"

Using one previous word instead of the whole history is called the Markov assumption. A trigram looks two words back, and an n-gram looks n − 1 words back. In the Berkeley Restaurant Project, a corpus of questions to a restaurant-finding system, P(want | I) = 0.33: a third of the time, "I" is followed by "want." SLP3, Chapter 3

These numbers capture real regularities. "want" is often followed by "to," which is English grammar; "I want" is common because people were asking for restaurants, which is knowledge about the task.

A language model can also give a whole sentence a probability. The chain rule multiplies the probability of each word given the words before it, and a bigram model shortens each context to one word:

P(<s> i want english food </s>) = P(i | <s>) × P(want | i) × P(english | want) × P(food | english) × P(</s> | food) = 0.25 × 0.33 × 0.0011 × 0.5 × 0.68 ≈ 0.000031

The number itself is tiny, so in practice models add log probabilities instead of multiplying, which avoids numerical underflow. What matters is the comparison: a more natural sentence gets a higher probability. That makes the model useful as a filter.

✎ reading noteReading note: the formula P(W1) × P(W2 | W1) × P(W3 | W1, W2) is the exact chain rule; a bigram model approximates each term with only the previous word, so P(W3 | W1, W2) becomes P(W3 | W2). "relles" should read "relies."

To write rather than score, the model samples. Picture the number line from 0 to 1 divided into intervals, one per word, each as long as that word's probability. Draw a random number between 0 and 1; the interval it lands in is the chosen word. Frequent words own large intervals and are picked often, rare words own tiny ones.

1. Start from <s> and sample the first word from the row for <s>, for example "the."
2. Move to the row for "the" and sample again, for example "big," then "red," then "dog."
3. Stop when the sample is </s>.

That is how a word probability table yields "The pink dog." or "The big red dog." The sentence may never have appeared in the training text: each pair of neighbors is plausible, but the whole is new. More context makes the output more coherent. Sentences sampled from a unigram model are word salad, bigram sentences have local coherence, and trigram or 4-gram sentences start to sound like their source. SLP3, Chapter 3

✎ reading noteReading note: the second thought bubble lists "fast" twice; each word appears once in a probability row, and the row's probabilities add up to 1.

Where counting runs out

Counting has two built-in limits. Any word pair that never occurred in the training text gets probability zero, so a single unseen pair makes a whole sentence impossible; smoothing methods such as add-one (Laplace) smoothing give unseen pairs a small share. And a short context window cannot connect words far apart in a sentence. Neural language models, the next topic, address both by learning representations instead of storing counts.

Word probabilities are also a fingerprint of who, or what, wrote a text. Language models favor certain words more than people do, so the frequency distribution of words in a body of text shifts when much of it is AI-written. Liang et al. (2024) at Stanford used this to estimate, across a whole corpus rather than per document, how much text had been substantially modified by large language models. In peer reviews for four AI conferences held after ChatGPT's release, the estimate was between 6.5% and 16.9%. Liang et al., ICML 2024

Stanford's free online course Probability for AI, taught by Chris Piech and Mehran Sahami, teaches the probability behind modern AI to learners who know basic algebra, with volunteer-led small sections. I plan to take it next and will write up what I learn. Probability for AI

The earlier notes covered how a computer can be designed to understand human language at the level of concepts: break words and sentences apart, sort and group them, then define and link the relations between them. With that, a computer can understand and process text.

But how do we interact with a computer that understands text, and how does it answer us? The technique behind it is something we meet all the time in everyday life: probability. Put simply, while the computer builds up its knowledge, it learns the probability of each word that follows another word. In "the big red dog," it has learned how likely red is after big and how likely dog is after red, and it chooses the next word from those learned chances.

That is why I want to learn probability properly. Stanford's free Probability for AI course is my next step, and I will summarize what I learn there.

The next question I want to explore: how do neural language models work?

This note is based on my study notes from the University of British Columbia (UBC) micro-certificate program "Natural Language Processing to Improve Patient Care," in the course "Overview of NLP and Large Language Models." Technical details were checked against Jurafsky and Martin, Speech and Language Processing (3rd edition draft, August 2026): Chapter 3, N-gram Language Models; and Liang et al. (2024), Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews.

Try it yourself

• Google Books Ngram Viewer: compare how often "big red" and "large red" appear in books over time. These are the same kinds of counts a bigram model starts from.
• Stanford Probability for AI: the free online course on the probability behind modern AI mentioned above.