先看左侧的词性标注，再看右侧的命名实体识别。红色部分是最需要分清的两点：模型里的 token 不等于一个字或一个词；实体可以是一个词组。底部是这一篇对我最重要的结论。

✎ 阅读提示根据我的手写笔记整理，做了几处修正和补充。

Start with POS tagging on the left, then NER on the right. The red notes mark the two points most worth keeping apart: a model token is not the same as a character or a word, and an entity can be a phrase. The note at the bottom is my main takeaway.

✎ 阅读提示Tidied up from my handwritten page, with a few corrections and additions.

可以把这一篇的内容理解为三步。它们是常见的处理环节，但不是每个系统都必须按同样顺序单独执行。

步骤	输入	输出	主要难点
分词 Tokenization	一段连续文字	一串 token	单位切多大：词、子词还是字符
句子切分 Sentence segmentation	一段文字	一个个句子	句号也可能属于缩写或数字
贴标签 Labeling	token 序列	每个 token 或片段的标签	同一个词在不同语境中可能对应不同标签

这一篇介绍两类基础标签任务：词性标注（part-of-speech tagging，POS tagging）和命名实体识别（named entity recognition，NER）。两者都属于序列标注（sequence labeling）：输入一串 token，为每个 token 输出一个标签，输出与输入等长。Sequence Labeling for Parts of Speech and Named Entities

✎ 阅读提示说明：课程和配图把这一过程称为“序列到序列”（sequence to sequence）。在 NLP 中，sequence-to-sequence 更常指机器翻译这类输入、输出长度可以不同的编码器–解码器任务；词性标注和 NER 更准确的名称是序列标注。

The topics in this note can be read as three steps. They are common stages, though not every system runs them separately or in the same order.

Step	Input	Output	Main difficulty
Tokenization	Running text	A sequence of tokens	Choosing the unit size: words, subwords, or characters
Sentence segmentation	Running text	Individual sentences	A period can also belong to an abbreviation or a number
Labeling	A token sequence	A label for each token or span	The same word can take different labels in different contexts

This note introduces two basic labeling tasks: part-of-speech (POS) tagging and named entity recognition (NER). Both are sequence labeling tasks: given a sequence of tokens, the system outputs one label per token, so the output has the same length as the input. Sequence Labeling for Parts of Speech and Named Entities

✎ 阅读提示Note: the course and infographics call this "sequence to sequence." In NLP, sequence-to-sequence usually refers to encoder–decoder tasks such as machine translation, where input and output lengths can differ. Sequence labeling is the more precise name for POS tagging and NER.

token 在 NLP 中有两种常见用法，容易混在一起。

用法	含义	例子
统计文本时：词型 vs 词例	词型（word type）是不重复的词；词例（word token / instance）是每一次出现	“the cat sat on the mat” 有 6 个词例、5 个词型（the 出现两次）
模型处理时：词元	分词器（tokenizer）切出的处理单位，模型读入的是一串词元编号	lower 可能被切成 low + er

第二种用法里，token 是模型读取文字的基本单位，不等于一个字或一个单词：它可以是完整的词、词的一部分，也可以是单个字符或标点。只按完整单词切分，会遇到训练中没见过的新词；只按字符切分，序列又会很长。子词（subword）分词是两者之间的折中。Words and Tokens

字节对编码（BPE）如何学到词元

字节对编码（byte-pair encoding，BPE）分两个阶段：

1. 学习阶段：从单个字符开始，反复统计语料中最常相邻出现的一对符号，把它们合并成新的词元，例如先合并 e + r → er，再合并 l + o → lo、lo + w → low。重复成千上万次，就得到一张词表。
2. 切分阶段：遇到新文本时，先拆成字符，再按学到的合并规则依次合并。即使训练中没见过 lower，也可以表示为 low + er。

实际的大模型词表通常有几万到二十万个词元。结果是常见词多为一个词元，罕见词和新词被拆成几个子词。不同语言的切分效率也不同：如果训练语料以英语为主，其他语言的同等内容可能需要更多词元。Words and Tokens

✎ 阅读提示读图说明：图题把分词和句子切分归在“文本归一化”下，这是教材中的一种组织方式。BPE 合并的依据是训练语料中的频率，切分新文本时按学到的顺序合并，不再重新统计新文本的频率。“GPT-4 的词元化”是示例，不同模型的分词器各不相同。

"Token" has two common uses in NLP, and they are easy to mix up.

Use	Meaning	Example
Counting a text: types vs tokens	A word type is a distinct word; a word token (or instance) is each occurrence	"the cat sat on the mat" has 6 word tokens but 5 types (the appears twice)
Inside a model	A unit produced by a tokenizer; the model reads a sequence of token IDs	lower might become low + er

In the second sense, a token is the basic unit a model reads. It is not necessarily a word or a character: it can be a whole word, part of a word, a single character, or punctuation. Splitting only into whole words leaves no way to handle words never seen in training; splitting only into characters makes sequences very long. Subword tokenization is the compromise between the two. Words and Tokens

How byte-pair encoding (BPE) learns tokens

Byte-pair encoding (BPE) works in two phases:

1. Training: start from single characters, find the pair of symbols that most often appear next to each other in the corpus, and merge them into a new token. For example, merge e + r into er, then l + o into lo, then lo + w into low. Repeating this thousands of times builds a vocabulary.
2. Encoding: split new text into characters, then apply the learned merges in order. Even if lower never appeared in training, it can be represented as low + er.

Vocabularies in large models typically hold tens of thousands to around 200,000 tokens. Common words usually become a single token, while rare and new words are split into several subwords. Efficiency also differs by language: when training text is mostly English, the same content in another language may need more tokens. Words and Tokens

✎ 阅读提示Reading note: "Coompromise" in the cloud is a typo for "compromise." BPE merges are driven by frequencies in the training corpus; when encoding new text, the learned merges are applied in order and the new text's frequencies play no role. The L + O + W diagram is a simplified picture of merges building a whole token from its parts.

句子切分（sentence segmentation）判断一段文字在哪里断句。问号和感叹号通常比较明确，句号则有歧义：它可以结束句子，也可以出现在 Dr.、Prof. 这类缩写或 19.99 这样的数字里。

常见做法是先分词，再判断每个句号属于词的一部分，还是句子边界。判断可以依靠规则（例如后面是否跟大写字母）、缩写词典，或者由机器学习模型根据上下文分类。Words and Tokens

例如 “Dr. Smith prescribed 2.5 mg at 8 a.m. The patient improved.” 中，Dr. 和 2.5 里的句点都不是句末；a.m. 的第二个句点既属于缩写，又标记了第一句的结束。临床记录中缩写很多，这类问题会更常见；切错句子会影响之后按句子进行的标注和抽取。

Sentence segmentation decides where one sentence ends and the next begins. Question marks and exclamation points are fairly reliable, but the period is ambiguous: it can end a sentence, mark an abbreviation such as Dr. or Prof., or sit inside a number such as 19.99.

A common approach is to tokenize first, then decide whether each period belongs to a word or marks a sentence boundary. That decision can rely on rules (for example, whether a capital letter follows), an abbreviation dictionary, or a machine learning classifier that uses context. Words and Tokens

Take "Dr. Smith prescribed 2.5 mg at 8 a.m. The patient improved." The periods in Dr. and 2.5 do not end a sentence, while the second period in a.m. both closes the abbreviation and ends the first sentence. Clinical notes are full of abbreviations, so this problem comes up often. A wrong split affects any later labeling or extraction done sentence by sentence.

词性标注（POS tagging）的输入是一串已切分的词 x₁, x₂, …, xₙ，输出是等长的标签序列 y₁, y₂, …, yₙ，每个标签对应一个词。例如 “The cat sat on the mat” 对应 限定词–名词–动词–介词–限定词–名词（DET NOUN VERB ADP DET NOUN）。

词性可以分为两类：

• 开放类：名词、动词、形容词、副词等实词，新词不断加入。
• 封闭类：冠词、介词、代词、连词等功能词，数量相对固定、出现频繁。

判断词性的线索

判断词性主要依靠三类线索：

• 词形：词缀提供提示，例如 -able 常出现在形容词中，-ly 常出现在副词中。
• 分布：周围出现什么词。冠词后面通常是名词或形容词。
• 先验概率：这个词在训练语料中最常被标成什么词性。

难点在于歧义。book 在 “Book that flight” 中是动词，在 “a good book” 中是名词。在英语语料中，有歧义的词只占不同词型的约 14–15%，但它们很常用，因此在实际文本里，约 55–67% 的词例存在词性歧义。SLP3 第 18 章

如何评估

词性标注用准确率（accuracy）评估：标对的词占全部词的比例。一个简单基线是总给每个词标它最常见的词性，准确率约 92%；现代模型在英语上约 97%，与人工标注的上限相近。混淆矩阵（confusion matrix）可以显示哪些词性最容易被互相混淆，帮助定位错误。SLP3 第 18 章

✎ 阅读提示读图说明：“序列到序列任务”更准确地说是序列标注。饼图中的 15% 和 60% 来自不同的计算基础——前者按不重复的词型计算，后者按文本中每次出现的词例计算——不是同一张饼图的两个部分。图中的“词形歧义”“词次覆盖”可以理解为词型歧义和词例歧义；“word? man?” 没有对应含义，可以忽略。

POS tagging takes a sequence of tokenized words x₁, x₂, …, xₙ and outputs a tag sequence y₁, y₂, …, yₙ of the same length, one tag per word. For example, "The cat sat on the mat" becomes DET NOUN VERB ADP DET NOUN (determiner, noun, verb, adposition, determiner, noun).

Parts of speech fall into two groups:

• Open classes: content words such as nouns, verbs, adjectives, and adverbs. New words join these classes all the time.
• Closed classes: function words such as articles, prepositions, pronouns, and conjunctions. Their membership is relatively fixed, and they are very frequent.

Clues for choosing a tag

Taggers rely on three kinds of clues:

• Word shape: affixes give hints, such as -able in adjectives or -ly in adverbs.
• Distribution: the neighboring words. A noun or adjective usually follows an article.
• Prior probability: the tag a word most often receives in the training data.

The difficulty is ambiguity. Book is a verb in "Book that flight" and a noun in "a good book." In English corpora, ambiguous words make up only about 14–15% of word types, but they are very common, so about 55–67% of word tokens in running text are ambiguous. SLP3, Chapter 18

Evaluation

POS taggers are evaluated by accuracy: the share of words tagged correctly. A simple baseline that always assigns each word its most frequent tag reaches about 92%. Modern taggers reach about 97% on English, close to the level of human agreement. A confusion matrix shows which tags are most often mistaken for each other, which helps locate errors. SLP3, Chapter 18

✎ 阅读提示Reading note: "Sequence to Sequence Mapping" is better called sequence labeling. The 15% and 60% figures are computed on different bases (distinct word types versus every word token in running text), so they are not two slices of one pie. The textbook gives about 92% for the most-frequent-tag baseline, slightly above the 91% shown.

命名实体识别（NER）要在文本中找出指称特定对象的片段，并判断它的类型。通用领域常见的类型包括人名（PER）、机构（ORG）、地点（LOC）和地缘政治实体（GPE），也常扩展到日期、金额等。医学文本中，任务通常会自定义药物、疾病、症状等类型。

与词性标注相比，NER 多了两个难点：

• 边界：实体不一定是一个词，可以是词组，例如 Jane Villanueva、community-acquired pneumonia。要判断从哪里开始、到哪里结束，而且文本中大多数词都不是实体。
• 类型歧义：Washington 可以是人名、球队（机构）、地点或政府（地缘政治实体）。

BIO 标注：把找片段变成逐词贴标签

BIO 标注让 NER 也可以按序列标注来做：B 表示实体开始，I 表示实体内部，O 表示不属于任何实体。下面是一个虚构的临床句子：

token	标签
Patient	O
started	O
amoxicillin	B-DRUG
for	O
community-acquired	B-DISEASE
pneumonia	I-DISEASE

这样，两个词组成的疾病名称被完整地标为一个实体。SLP3 第 18 章

标签体系由任务决定

在我的手写笔记里，duck、dog、cat 被归为 Animal，milk、cola 被归为 Drink。这是按语义类别贴标签。通用 NER 主要识别专有名称，一般不会把普通名词 dog 标成实体；但只要任务定义了相应类型，标签体系也可以覆盖普通名词——医学 NER 中的 pneumonia、high fever 就不是专有名称。因此，先定义“要识别什么、分成哪几类”，是做 NER 的第一步。

方法与评估

实现 NER 的方法包括规则与词典、监督式机器学习、神经序列模型，以及大语言模型（large language models，LLM）。选择时需要在准确性、效率和可获得的标注数据之间权衡。

评估有两种粒度：

粒度	计算单位	特点
逐 token（per-token）	每个词的标签是否正确	大多数词标 O，数值容易显得很高
按实体（课程称 per-field）	整个实体的边界和类型是否都正确	常用精确率、召回率和 F1

如果系统只把 Jane 标为人名，漏掉 Villanueva，按实体评估就不算识别正确。NER 通常以整个实体为单位，计算精确率（precision）、召回率（recall）和 F1。SLP3 第 18 章

✎ 阅读提示读图说明：右下角两个评估指标的说明不准确。“逐词准确率”衡量每个词的标签是否正确，不等于实体识别的完整性；“整场精确率”应理解为按实体计算的精确率，即系统标出的实体中，边界和类型都正确的比例。图中 Washington 的“组织”可以是球队，政府含义通常归为地缘政治实体。

NER finds spans of text that refer to specific things and assigns each one a type. General-domain types include person (PER), organization (ORG), location (LOC), and geo-political entity (GPE), often extended to dates and amounts of money. In medical text, tasks usually define their own types, such as drug, disease, and symptom.

Compared with POS tagging, NER adds two difficulties:

• Boundaries: an entity need not be one word; it can be a phrase such as Jane Villanueva or community-acquired pneumonia. The system must decide where each span starts and ends, and most words in a text are not entities at all.
• Type ambiguity: Washington can be a person, a sports team (organization), a location, or a government (geo-political entity).

BIO tagging: turning span-finding into word-by-word labels

BIO tagging lets NER be treated as sequence labeling. B marks the beginning of an entity, I marks a token inside an entity, and O marks a token outside any entity. Here is a made-up clinical sentence:

Token	Label
Patient	O
started	O
amoxicillin	B-DRUG
for	O
community-acquired	B-DISEASE
pneumonia	I-DISEASE

The two-word disease name is labeled as one complete entity. SLP3, Chapter 18

The task defines the labels

In my handwritten notes, duck, dog, and cat are grouped as Animal, and milk and cola as Drink. That is labeling by semantic category. General-domain NER mainly targets proper names and would not normally tag a common noun like dog. Still, a task can define types that cover common nouns: pneumonia and high fever in medical NER are not proper names. Deciding what to find and how to categorize it is the first step in any NER project.

Methods and evaluation

NER can be done with rules and dictionaries, supervised machine learning, neural sequence models, or large language models (LLMs). The choice involves trade-offs among accuracy, efficiency, and the labeled data available.

Evaluation can work at two levels:

Level	Unit	What to know
Per token	Whether each word's label is correct	Most words are O, so scores can look high
Per entity (the course calls it per-field)	Whether an entity's boundaries and type are both correct	Usually reported as precision, recall, and F1

If a system tags Jane as a person but misses Villanueva, entity-level evaluation counts that as an error. NER is usually scored with the whole entity as the unit, using precision, recall, and F1. SLP3, Chapter 18

✎ 阅读提示Reading note: "Harmoxylin" is not a real drug name; amoxicillin is a real antibiotic. In the Washington example, the second "Organization" label (the Capitol) would usually be a geo-political entity. "Field Precision" here means entity-level precision: of the entities the system labels, the share whose boundaries and type are both correct.

这一期最需要掌握的概念是 token。token 是 AI 阅读和理解文字的基本单位，它不等于一个字或一个单词，而是把文字切成一小块一小块、AI 能处理的单位：可以是一个词组、一个片段，也可以是一个词的一部分。分词（tokenization）就是完成一段文字词元划分的过程。

我的理解是，词元划分越有效，模型学习时的效率就越高，也越节省算力，所以有很多研究专门探索如何进行更有效的词元划分。

有了 token 之后，对文本的处理可以简单地概括为“贴标签”。标签可以有很多种：比如词性，动词、名词、形容词；再比如按词义划分，狗、猫、鸡、鸭都是动物，或者狗和猫是哺乳动物，鸡和鸭是禽类。贴标签是学习的第一步，方便之后进行知识分类和知识体系的构建。

在实际应用中，我需要特别注意“标签”。既然最基础的一步是对内容分类，而分类可以有不同的类别和方法，那么设计标签体系，就是我想构建自己的模型时，可以加入自己经验和对内容理解的部分。

下一步我想研究的问题是：如何从词义走向知识网络？

The most important concept in this note is the token. A token is the basic unit AI uses to read and process text. It is not the same as a character or a word; text is cut into small pieces that a model can work with, and a piece can be a phrase, a fragment, or part of a word. Tokenization is the process of dividing a passage into these units.

As I understand it, more efficient tokenization makes model training more efficient and saves computing power. That is why a lot of research focuses specifically on better ways to tokenize.

Once text is tokenized, much of what follows can be summed up as labeling. Labels come in many kinds: parts of speech, such as verb, noun, and adjective, or meaning-based groups, such as dog, cat, chicken, and duck as animals, or dogs and cats as mammals and chickens and ducks as poultry. Labeling is the first step of learning. It makes it easier to classify knowledge and to build a knowledge structure later.

In practice, I need to pay close attention to labels. If the most basic step is classifying content, and classification can use different categories and methods, then designing the label set is where I can bring in my own experience and understanding of the content when building my own model.

The next question I want to explore: how do we move from word meaning to a knowledge network?

✎ 阅读提示整理后的版本改了这些地方：把“序列到序列”改为序列标注；把“词意分类”改为按实体类型归类，并注明语义类别与通用 NER 的区别；补充了 BIO 标注和按实体评估。另外，笔记中的 “Name Entity Recognition” 应为 Named Entity Recognition。

✎ 阅读提示What changed in the tidied version: "sequence to sequence" becomes sequence labeling; "meaning categories" becomes grouping entities by type, with a note on how semantic categories differ from general-domain NER; BIO tagging and entity-level evaluation are added. The notes also wrote "Name Entity Recognition"; it should read Named Entity Recognition.

本篇由 UBC “Natural Language Processing to Improve Patient Care” 微证书课程学习笔记整理而成，母稿记录的课程为 “Overview of NLP and Large Language Models”。技术核对使用 Jurafsky 与 Martin 的 《Speech and Language Processing》（第 3 版草稿，2026 年 8 月）第 2 章 Words and Tokens 和第 18 章 Sequence Labeling for Parts of Speech and Named Entities。

动手探索

想看看一句话会被切成哪些词元，可以使用 Tiktokenizer：输入一段中文或英文，查看词元划分和数量。可以比较同一句话的中英文版本各需要多少词元，或者试试医学术语会被怎样拆分。

This note is based on my study notes from the University of British Columbia (UBC) micro-certificate program "Natural Language Processing to Improve Patient Care," in the course "Overview of NLP and Large Language Models." Technical details were checked against Jurafsky and Martin, Speech and Language Processing (3rd edition draft, August 2026): Chapter 2, Words and Tokens, and Chapter 18, Sequence Labeling for Parts of Speech and Named Entities.

Try it yourself

To see how a sentence is split into tokens, try Tiktokenizer. Paste in English or Chinese text to see the token boundaries and count. Compare how many tokens the same sentence needs in each language, or see how a medical term is split.