Advanced Text Processing: Tokens, Sentences, and Labels

Start with POS tagging on the left, then NER on the right. The red notes mark the two points most worth keeping apart: a model token is not the same as a character or a word, and an entity can be a phrase. The note at the bottom is my main takeaway.

The topics in this note can be read as three steps. They are common stages, though not every system runs them separately or in the same order.

Step	Input	Output	Main difficulty
Tokenization	Running text	A sequence of tokens	Choosing the unit size: words, subwords, or characters
Sentence segmentation	Running text	Individual sentences	A period can also belong to an abbreviation or a number
Labeling	A token sequence	A label for each token or span	The same word can take different labels in different contexts

This note introduces two basic labeling tasks: part-of-speech (POS) tagging and named entity recognition (NER). Both are sequence labeling tasks: given a sequence of tokens, the system outputs one label per token, so the output has the same length as the input. Sequence Labeling for Parts of Speech and Named Entities

✎ reading noteNote: the course and infographics call this "sequence to sequence." In NLP, sequence-to-sequence usually refers to encoder–decoder tasks such as machine translation, where input and output lengths can differ. Sequence labeling is the more precise name for POS tagging and NER.

"Token" has two common uses in NLP, and they are easy to mix up.

Use	Meaning	Example
Counting a text: types vs tokens	A word type is a distinct word; a word token (or instance) is each occurrence	"the cat sat on the mat" has 6 word tokens but 5 types (the appears twice)
Inside a model	A unit produced by a tokenizer; the model reads a sequence of token IDs	lower might become low + er

In the second sense, a token is the basic unit a model reads. It is not necessarily a word or a character: it can be a whole word, part of a word, a single character, or punctuation. Splitting only into whole words leaves no way to handle words never seen in training; splitting only into characters makes sequences very long. Subword tokenization is the compromise between the two. Words and Tokens

How byte-pair encoding (BPE) learns tokens

Byte-pair encoding (BPE) works in two phases:

1. Training: start from single characters, find the pair of symbols that most often appear next to each other in the corpus, and merge them into a new token. For example, merge e + r into er, then l + o into lo, then lo + w into low. Repeating this thousands of times builds a vocabulary.
2. Encoding: split new text into characters, then apply the learned merges in order. Even if lower never appeared in training, it can be represented as low + er.

Vocabularies in large models typically hold tens of thousands to around 200,000 tokens. Common words usually become a single token, while rare and new words are split into several subwords. Efficiency also differs by language: when training text is mostly English, the same content in another language may need more tokens. Words and Tokens

✎ reading noteReading note: "Coompromise" in the cloud is a typo for "compromise." BPE merges are driven by frequencies in the training corpus; when encoding new text, the learned merges are applied in order and the new text's frequencies play no role. The L + O + W diagram is a simplified picture of merges building a whole token from its parts.

Sentence segmentation decides where one sentence ends and the next begins. Question marks and exclamation points are fairly reliable, but the period is ambiguous: it can end a sentence, mark an abbreviation such as Dr. or Prof., or sit inside a number such as 19.99.

A common approach is to tokenize first, then decide whether each period belongs to a word or marks a sentence boundary. That decision can rely on rules (for example, whether a capital letter follows), an abbreviation dictionary, or a machine learning classifier that uses context. Words and Tokens

Take "Dr. Smith prescribed 2.5 mg at 8 a.m. The patient improved." The periods in Dr. and 2.5 do not end a sentence, while the second period in a.m. both closes the abbreviation and ends the first sentence. Clinical notes are full of abbreviations, so this problem comes up often. A wrong split affects any later labeling or extraction done sentence by sentence.

POS tagging takes a sequence of tokenized words x₁, x₂, …, xₙ and outputs a tag sequence y₁, y₂, …, yₙ of the same length, one tag per word. For example, "The cat sat on the mat" becomes DET NOUN VERB ADP DET NOUN (determiner, noun, verb, adposition, determiner, noun).

Parts of speech fall into two groups:

• Open classes: content words such as nouns, verbs, adjectives, and adverbs. New words join these classes all the time.
• Closed classes: function words such as articles, prepositions, pronouns, and conjunctions. Their membership is relatively fixed, and they are very frequent.

Clues for choosing a tag

Taggers rely on three kinds of clues:

• Word shape: affixes give hints, such as -able in adjectives or -ly in adverbs.
• Distribution: the neighboring words. A noun or adjective usually follows an article.
• Prior probability: the tag a word most often receives in the training data.

The difficulty is ambiguity. Book is a verb in "Book that flight" and a noun in "a good book." In English corpora, ambiguous words make up only about 14–15% of word types, but they are very common, so about 55–67% of word tokens in running text are ambiguous. SLP3, Chapter 18

Evaluation

POS taggers are evaluated by accuracy: the share of words tagged correctly. A simple baseline that always assigns each word its most frequent tag reaches about 92%. Modern taggers reach about 97% on English, close to the level of human agreement. A confusion matrix shows which tags are most often mistaken for each other, which helps locate errors. SLP3, Chapter 18

✎ reading noteReading note: "Sequence to Sequence Mapping" is better called sequence labeling. The 15% and 60% figures are computed on different bases (distinct word types versus every word token in running text), so they are not two slices of one pie. The textbook gives about 92% for the most-frequent-tag baseline, slightly above the 91% shown.

NER finds spans of text that refer to specific things and assigns each one a type. General-domain types include person (PER), organization (ORG), location (LOC), and geo-political entity (GPE), often extended to dates and amounts of money. In medical text, tasks usually define their own types, such as drug, disease, and symptom.

Compared with POS tagging, NER adds two difficulties:

• Boundaries: an entity need not be one word; it can be a phrase such as Jane Villanueva or community-acquired pneumonia. The system must decide where each span starts and ends, and most words in a text are not entities at all.
• Type ambiguity: Washington can be a person, a sports team (organization), a location, or a government (geo-political entity).

BIO tagging: turning span-finding into word-by-word labels

BIO tagging lets NER be treated as sequence labeling. B marks the beginning of an entity, I marks a token inside an entity, and O marks a token outside any entity. Here is a made-up clinical sentence:

Token	Label
Patient	O
started	O
amoxicillin	B-DRUG
for	O
community-acquired	B-DISEASE
pneumonia	I-DISEASE

The two-word disease name is labeled as one complete entity. SLP3, Chapter 18

The task defines the labels

In my handwritten notes, duck, dog, and cat are grouped as Animal, and milk and cola as Drink. That is labeling by semantic category. General-domain NER mainly targets proper names and would not normally tag a common noun like dog. Still, a task can define types that cover common nouns: pneumonia and high fever in medical NER are not proper names. Deciding what to find and how to categorize it is the first step in any NER project.

Methods and evaluation

NER can be done with rules and dictionaries, supervised machine learning, neural sequence models, or large language models (LLMs). The choice involves trade-offs among accuracy, efficiency, and the labeled data available.

Evaluation can work at two levels:

Level	Unit	What to know
Per token	Whether each word's label is correct	Most words are O, so scores can look high
Per entity (the course calls it per-field)	Whether an entity's boundaries and type are both correct	Usually reported as precision, recall, and F1

If a system tags Jane as a person but misses Villanueva, entity-level evaluation counts that as an error. NER is usually scored with the whole entity as the unit, using precision, recall, and F1. SLP3, Chapter 18

✎ reading noteReading note: "Harmoxylin" is not a real drug name; amoxicillin is a real antibiotic. In the Washington example, the second "Organization" label (the Capitol) would usually be a geo-political entity. "Field Precision" here means entity-level precision: of the entities the system labels, the share whose boundaries and type are both correct.

The most important concept in this note is the token. A token is the basic unit AI uses to read and process text. It is not the same as a character or a word; text is cut into small pieces that a model can work with, and a piece can be a phrase, a fragment, or part of a word. Tokenization is the process of dividing a passage into these units.

As I understand it, more efficient tokenization makes model training more efficient and saves computing power. That is why a lot of research focuses specifically on better ways to tokenize.

Once text is tokenized, much of what follows can be summed up as labeling. Labels come in many kinds: parts of speech, such as verb, noun, and adjective, or meaning-based groups, such as dog, cat, chicken, and duck as animals, or dogs and cats as mammals and chickens and ducks as poultry. Labeling is the first step of learning. It makes it easier to classify knowledge and to build a knowledge structure later.

In practice, I need to pay close attention to labels. If the most basic step is classifying content, and classification can use different categories and methods, then designing the label set is where I can bring in my own experience and understanding of the content when building my own model.

The next question I want to explore: how do we move from word meaning to a knowledge network?

This note is based on my study notes from the University of British Columbia (UBC) micro-certificate program "Natural Language Processing to Improve Patient Care," in the course "Overview of NLP and Large Language Models." Technical details were checked against Jurafsky and Martin, Speech and Language Processing (3rd edition draft, August 2026): Chapter 2, Words and Tokens, and Chapter 18, Sequence Labeling for Parts of Speech and Named Entities.

Try it yourself

To see how a sentence is split into tokens, try Tiktokenizer. Paste in English or Chinese text to see the token boundaries and count. Compare how many tokens the same sentence needs in each language, or see how a medical term is split.