External Evaluation of 3 Commercial Artificial Intelligence Algorithms for Independent Assessment of Screening Mammograms

方法图解Visual method notes

研究背景Background

问题Problem

人群乳腺 X 线筛查使乳腺癌死亡率下降了 20%–40%，但放射科医生工作量大，不同阅片者的判读质量差异明显。

Population mammography screening has cut breast cancer mortality by 20% to 40%, but radiologist workload is high and reading quality varies between readers.

研究空白Gap

已有深度学习计算机辅助检测（CAD）研究报告了接近放射科医生的水平，但常常使用非筛查人群、对照医生表现偏弱，或算法并未上市；而且没有一项研究由第三方对多款相互竞争的商用算法做外部验证和比较。

Deep-learning CAD studies reported radiologist-level performance, but they often used non-screening populations, weak human comparators, or algorithms not on the market — and none had a third party externally validate and compare competing commercial algorithms.

研究目标Objective

对 3 款已商用的 AI CAD 算法作为独立阅片者进行外部评估，并评估它们与放射科医生组合时的筛查表现。

To externally evaluate 3 commercially available AI CAD algorithms as independent mammography readers, and to assess screening performance when they are combined with radiologists.

研究方法Methods

研究速览Study at a glance

方法分步讲解Step by step

1. 1

   构建测试集 · Build the test set

   输入：斯德哥尔摩筛查队列（CSAW）。团队纳入筛查后 12 个月内确诊乳腺癌的全部女性（“病例”），再随机抽取随访 2 年无癌的健康女性（“对照”），最终 8805 人：739 例病例、8066 名对照。这就像出一份带标准答案的考卷：每位女性是否真有癌，由病理和登记库随访确认，而不是由任何阅片者决定。与真实筛查日不同，这里癌症被刻意“放大”（病例对照抽样，C5），所以后面几步必须校正。

   Input: a Stockholm screening cohort (CSAW). The team took every woman diagnosed with breast cancer within 12 months after a screening exam (the 'cases'), plus a random draw of healthy women who stayed cancer-free for 2 years (the 'controls'), ending with 8805 women: 739 cases and 8066 controls. It is like building an exam paper with a known answer key: whether each woman truly had cancer was confirmed by pathology and registry follow-up, not by any reader. Unlike a real screening day, cancers are deliberately over-represented (case-control sampling, C5), so later steps must correct for that.

   为什么这样做Why this step

   在给任何人打分之前，先要有一份独立于 AI 和医生的可靠标准答案。

   A reliable answer key, independent of both the AI and the radiologists, is needed before anyone can be graded.

   技术细节Technical detail · M1, M3, M6, M7, M9, M10, M11, M15

   来自 CSAW（卡罗林斯卡，2008–2015，Hologic 全视野数字乳腺摄影）的回顾性病例对照样本：病例 = 筛查时或 12 个月内病理确诊（排除 >12 个月的 419 次检查）；对照 = 随机 10 000 名健康女性经排除（要求 2 年无癌随访）；最终 N = 8805（739/8066）。

   Retrospective case-control sample from CSAW (Karolinska, 2008–2015, Hologic FFDM): cases = pathology-confirmed cancer at screening or ≤12 months (419 exams >12 months excluded); controls = random 10 000 healthy women with exclusions (2-year cancer-free follow-up required); final N = 8805 (739/8066).

   C2 · 乳腺 X 线筛查与双人阅片（double reading） · Screening mammography with double readingC4 · 外部验证（external validation） · External validationC5 · 病例对照（富集）抽样与患病率 · Case-control (enriched) sampling and prevalence
2. 2

   调出医生的真实判读 · Retrieve radiologists' real decisions

   输入：筛查登记库。团队调出每份检查当年医生的真实判断：一读、二读（通常更有经验，且能看到一读结果）以及会诊决定。每个判断都是简单的“正常/异常”，所以可以用敏感度和特异度（C1）来打分。可以把它想成直接使用医生“当年已交的答卷”，而不是请他们重新考一次。

   Input: the screening register. For each exam the team pulled what the radiologists actually decided at the time, years earlier: first reader, second reader (often more experienced and able to see the first reader's call), and the consensus meeting. Each decision is a simple normal/abnormal, so it can be scored with sensitivity and specificity (C1). Think of it as using the human doctors' already-submitted exam answers rather than asking them to retake the test.

   为什么这样做Why this step

   真实临床决策提供了贴近现实的人类基准，而不是实验室阅片研究。

   Real-world clinical decisions give a realistic human benchmark instead of a lab reader study.

   技术细节Technical detail · M12, M13, M14

   从斯德哥尔摩-哥特兰区域癌症中心登记库提取 25 位一读、20 位二读及会诊（正常/召回）的前瞻性二分类判读，通过个人身份号码关联。

   Prospectively recorded binary assessments of 25 first readers, 20 second readers and consensus (normal/recall) from the Regional Cancer Center Stockholm-Gotland register, linked via personal identity numbers.

   C1 · 敏感度与特异度（sensitivity / specificity） · Sensitivity and specificityC2 · 乳腺 X 线筛查与双人阅片（double reading） · Screening mammography with double reading
3. 3

   三款商用 AI 盲测 · Run three commercial AIs blind

   输入：只有乳腺 X 线图像。三家厂商的算法（AI-1、AI-2、AI-3）在研究者自己的电脑上运行；厂商从未看到图像，也不参与分析——这就是外部验证（C4）。每个 AI 给每侧乳房打 0–1 的可疑度分，两侧中较高的分作为这位女性的分数。就像三名外校学生参加同一场封闭考试，考场里没有他们学校的人。

   Input: the mammogram images only. The three vendor algorithms (AI-1, AI-2, AI-3) were run on the researchers' own computers; the companies never saw the images or took part in the analysis — this is external validation (C4). Each AI gives every breast a suspicion score from 0 to 1, and the higher of the two breasts becomes the woman's score. It is like three outside students sitting the same sealed exam, with nobody from their school in the room.

   为什么这样做Why this step

   防止厂商针对测试数据做调整，使评分反映产品面对新病人时的真实表现。

   Keeps vendors from tuning to the test data, so the scores reflect how the products would really perform on new patients.

   技术细节Technical detail · M2, M16, M17, M18

   3 款匿名商用深度学习 CAD 产品（未获 FDA 独立阅片批准）在本地仅用图像推理；每侧乳房评分 0–1；检查层面评分 = max(左, 右)。

   Images-only inference, locally, by 3 anonymous commercial deep-learning CAD products (not FDA-approved as independent readers); per-breast score 0–1; exam-level score = max(left, right).

   C3 · 深度学习计算机辅助检测（AI CAD） · Deep-learning computer-aided detection (AI CAD)C4 · 外部验证（external validation） · External validation
4. 4

   用 AUC 检查排序能力 · Rank-quality check with AUC

   输入：每个 AI 的评分加上标准答案。在选任何切点之前，团队先看每个 AI 把患癌女性排在未患癌女性前面的能力，用 AUC（C7）概括，包括总体和亚组（年龄、乳腺密度、癌症是在筛查时发现还是之后在临床上发现）。DeLong 检验（C9）判断一个 AI 的 AUC 是否真的高于另一个。就像先比较三位评委给选手排名次的水平，再决定及格线放在哪里。

   Input: each AI's scores plus the answer key. Before picking any cut-off, the team asked how well each AI ranks women with cancer above women without, summarized as the AUC (C7), both overall and in subgroups (age, breast density, and whether the cancer was found at screening or later as a clinically detected cancer). The DeLong test (C9) checks whether one AI's AUC is really higher than another's. It is like comparing how well three judges order contestants, before deciding where the pass mark goes.

   为什么这样做Why this step

   AUC 不依赖某一个阈值，可以公平地比较各个 AI。

   AUC compares the AIs fairly without depending on any single threshold.

   技术细节Technical detail · M8, M19, M20, M25

   每个算法计算总体 AUC，以及按年龄、乳腺密度（4 个体位均值，以中位数划分）和检出方式分组的 AUC；两两 AUC 差异用 DeLong 检验；置信区间用三明治方差估计量；次要分析加入筛查后 12–23 个月确诊的 174 例癌症。

   AUC per algorithm overall and by age, mammographic density (4-view mean, split at median) and detection mode; pairwise AUC differences by DeLong test; CIs by sandwich variance estimator; secondary analysis adding 174 cancers diagnosed 12–23 months after screening.

   C7 · ROC 曲线与曲线下面积（AUC） · ROC curve and AUCC9 · DeLong 检验 · DeLong test
5. 5

   让 AI 与医生同样严格 · Match AI to radiologists' strictness

   输入：AI 的连续评分和一读医生的特异度。医生给出“是/否”，而 AI 给出一个数字，所以团队为每个 AI 选了一个切点（C6），让它与一读医生一样“严格”——把约 96.6% 的健康女性判为正常。然后才能问一个公平的问题：在误报水平相同的情况下，谁抓到的癌更多？就像先把两个烟雾报警器的误报率调成一样，再看哪个能发现更多真实火情。

   Input: continuous AI scores and the first readers' specificity. Doctors give yes/no, but the AI gives a number, so the team picked a cut-off (C6) for each AI that makes it as 'strict' as the first-reader radiologists — calling about 96.6% of healthy women normal. Then they could ask the fair question: at the same false-alarm level, who catches more cancers? It is like setting two smoke detectors to the same false-alarm rate before seeing which one detects more real fires.

   为什么这样做Why this step

   如果没有统一的工作点，更高的敏感度可能只是因为误报更多。

   Without a common operating point, a higher sensitivity could simply mean more false alarms.

   技术细节Technical detail · M21, M23

   每个评分在特异度最接近一读 96.6% 的切点处二分类；指标：敏感度、特异度、准确率、异常判读率、癌症检出率、假阴性率（每 1000 人）和 PPV，对每个 AI 及一读、二读和会诊分别计算。

   Each score dichotomized at the cut point giving specificity closest to first readers' 96.6%; metrics: sensitivity, specificity, accuracy, abnormal interpretation rate, cancer detection rate, false-negative rate (per 1000) and PPV, for each AI and for first reader, second reader and consensus.

   C1 · 敏感度与特异度（sensitivity / specificity） · Sensitivity and specificityC6 · 评分阈值（工作点，operating point） · Score threshold (operating point)
6. 6

   按真实患病率重新加权 · Re-weight to real screening prevalence

   输入：富集后的样本。因为癌症被放大了（S1），像“每 1000 名筛查女性检出几例癌”这样的数字会失真。团队用分层自助重抽样（C8）：1000 次分别按 14:1 的比例重新抽取健康女性和癌症病例，目标是模拟真实筛查人群（约 0.5% 为筛查检出癌）。就像把样本重新调配，让比例重新接近真实人群——但它只能重复使用研究中已有的女性，不能加入新类型的人。

   Input: the enriched sample. Because cancers were over-represented (S1), numbers like 'cancers found per 1000 women screened' would be misleading. The team used stratified bootstrap resampling (C8): 1000 times they redrew healthy women and cancer cases separately at a 14:1 ratio, aiming to mimic the real screening population (about 0.5% screen-detected cancer). It is like re-mixing a sample so the proportions look like the real crowd again — but it can only reuse the women already in the study, not add new kinds of women.

   为什么这样做Why this step

   依赖患病率的指标（每 1000 人比率、PPV）只有在真实的病例构成下才有意义。

   Prevalence-dependent metrics (rates per 1000, PPV) are only meaningful at a realistic case mix.

   技术细节Technical detail · M22

   分层 bootstrap，1000 次抽样，健康:癌症 = 14:1，以近似原始队列（约 0.5% 为筛查检出癌）。

   Stratified bootstrapping, 1000 samples, healthy:cancer = 14:1, to approximate the source cohort (~0.5% screen-detected cancer).

   C5 · 病例对照（富集）抽样与患病率 · Case-control (enriched) sampling and prevalenceC8 · 自助重抽样（bootstrap） · Bootstrap resamplingC1 · 敏感度与特异度（sensitivity / specificity） · Sensitivity and specificity
7. 7

   测试 AI 与医生搭档 · Test AI–radiologist teamwork

   输入：各 AI 和医生的“是/否”判断。团队模拟了新的阅片组合：每个 AI 与一读、二读或两位医生搭配（任一方标记即为异常）；三个 AI 联合（任一 AI 标记即异常）；全部 AI 加医生（至少 2 方标记才算异常）。对每种组合统计检出的癌症数与异常判读数（召回工作量）。就像测试把两人小组中的一个人换成机器，能否发现更多问题、又要付出多少误报代价。

   Input: the yes/no calls from the AIs and the readers. The team simulated new reading teams: each AI paired with the first reader, second reader or both (abnormal if anyone flags it); all three AIs together (abnormal if any AI flags); and all AIs plus radiologists (abnormal if at least 2 flag). For each team they counted cancers found against abnormal calls (recall workload). It is like testing whether swapping one human member of a two-person team for a machine catches more problems, and at what cost in false alarms.

   为什么这样做Why this step

   现实中 AI 最可能替代或辅助其中一位阅片者，所以关键问题是组合后的表现，而不只是 AI 单独的表现。

   In practice AI would most likely replace or support one reader, so the useful question is how combined teams perform, not just AI alone.

   技术细节Technical detail · M24

   用“或”规则（≥1 方异常）模拟 AI + 一读/二读/双读以及 3 个 AI 联合，用“N 中至少 2 方”规则模拟全部 AI 加医生；比较异常判读数与检出癌数。

   Simulated combinations using OR-rules (≥1 abnormal) for AI + first/second/both readers and for all 3 AIs, and a ≥2-of-N rule for all AIs plus radiologists; compared abnormal interpretations vs cancers detected.

   C2 · 乳腺 X 线筛查与双人阅片（double reading） · Screening mammography with double readingC6 · 评分阈值（工作点，operating point） · Score threshold (operating point)

读懂方法需要的前置知识Concepts you need first

研究结果Results

• 3 款算法区分癌症的能力都不错，但并不相同：AI-1 的 AUC 为 0.956，AI-2 为 0.922，AI-3 为 0.920（AI-1 更高，P < .001）。在各年龄、乳腺密度和检出方式亚组中，AI-1 都保持最好。 p. 3–4

  All 3 algorithms ranked cancers well, but not equally: AUC 0.956 for AI-1 versus 0.922 for AI-2 and 0.920 for AI-3 (AI-1 higher, P < .001). AI-1 stayed best in every age, density and detection-mode subgroup. p. 3–4
• 在较年轻女性（<55 岁）和致密型乳腺中表现下降，对筛查阴性后 12 个月内临床发现的间期癌下降最明显：AI-1 对间期癌的 AUC 为 0.810，对筛查检出癌为 0.984。 p. 3–4

  Performance dropped for younger women (<55 years) and denser breasts, and especially for cancers that surfaced clinically within 12 months after a negative screen: AI-1 AUC 0.810 for these interval cancers versus 0.984 for screen-detected ones. p. 3–4
• 在 113 663 例检查的模拟筛查人群中，把特异度固定在第一阅片医生的 96.6% 时，AI-1、AI-2、AI-3 的敏感度分别为 81.9%、67.0%、67.4%；第一阅片医生为 77.4%（AI-1 更高，P = .03），第二阅片医生为 80.1%，会诊（consensus）决定为 85.0%（与 AI-1 差异无统计学意义）。 p. 4

  In a simulated screening population of 113 663 exams, at the first readers' 96.6% specificity, sensitivity was 81.9% for AI-1, 67.0% for AI-2 and 67.4% for AI-3, versus 77.4% for the first reader (AI-1 higher, P = .03), 80.1% for the second reader and 85.0% for the consensus decision (no significant difference from AI-1). p. 4
• 第一阅片医生与 AI-1 组合（任一方判为异常即异常）的敏感度为 88.6%、特异度为 93.0%：比第一阅片医生单独多检出 15% 的癌症（加入第二阅片医生只多 12%），但异常判读数增加了 78%（加入第二阅片医生只增加 24%）。 p. 4–5

  Pairing the first reader with AI-1 (abnormal if either flags) reached 88.6% sensitivity at 93.0% specificity — 15% more cancers than the first reader alone, versus 12% more when the second reader was added — but abnormal interpretations rose by 78%, versus 24%. p. 4–5
• 再叠加更多算法收益有限：3 款 AI 合用（任一判阳即阳）的敏感度为 86.7%，但特异度只有 92.5%；3 款 AI 加 2 名阅片医生的 5 方投票，也没有明显优于 AI-1 加第一阅片医生。 p. 4–5

  Adding more algorithms gave little extra: all 3 AIs together (positive if any flags) reached 86.7% sensitivity but only 92.5% specificity, and a 5-way vote of 3 AIs plus 2 readers did not clearly beat AI-1 plus the first reader. p. 4–5

关键数字Key numbers

效应量与置信区间Estimates with confidence intervals

研究局限Limitations

研究设计Study design · 1

• 样本按病例对照方式富集了癌症；筛查层面的指标靠重加权模拟，并非在连续队列中观察到。

  Cancer-enriched case-control sample; screening-level figures are simulated by re-weighting, not observed in a consecutive cohort.
  p. 6

方法学Methodology · 1

• 算法看不到既往片子、激素用药和症状，而放射科医生可以。

  The algorithms saw no prior mammograms, hormone use or symptoms, unlike the radiologists.
  p. 6

数据Data · 1

• 排除了有乳房假体或既往乳腺癌的女性。

  Women with implants or prior breast cancer were excluded.
  p. 6

外推性Generalizability · 2

• 仅一家瑞典医院、一种设备（Hologic）；欧洲的阅片标准与美国不同。

  One Swedish hospital and one equipment vendor (Hologic); European reading standards differ from the US.
  p. 5–6

• 被测算法版本已被替换，厂商保持匿名。

  The tested algorithm versions have since been replaced, and vendors remain anonymous.
  p. 2, 6

解读Interpretation · 1

• AI 加一名阅片医生会大幅增加异常判读；仍需前瞻性试验和成本效益分析。

  AI plus a reader sharply increases abnormal calls; prospective trials and cost-benefit analysis are still needed.
  p. 5–6

我的思考My thoughts

整体评价Overall assessment

我认为还可能存在的不足Further limitations I see

• 参考标准依赖放射科医生：两位医生都漏掉、12–24 个月后才发现的癌症被排除在外，AI 可能更早发现的病例看不到。

  The reference standard leans on the radiologists: cancers both readers missed that surfaced 12–24 months later are left out, so possible earlier AI detection goes unseen.
  p. 2, 4
• 每款 AI 的阈值都是在测试数据上调定的；预先固定的阈值表现可能不同。

  Each AI's threshold was tuned on the test data itself; a threshold fixed in advance may perform differently.
  p. 2–3
• 部分数字对不上（对照 7971 与 8066；739 例癌症中 14 例未分类），样本流程难以复现。

  Some counts do not add up (7971 vs 8066 controls; 14 of 739 cancers unclassified), so the sample flow is hard to reproduce.
  p. 2–3

优先考虑的改进Most feasible improvement

• 把 AI-1 以静默模式运行在之后一年的连续筛查检查上，阈值沿用本研究的固定值，并对每位女性随访 24 个月。报告敏感度、特异度、召回量，以及 AI 提前标记了多少后来才发现的癌症。

  Run AI-1 silently, with this study's threshold frozen, on one later year of consecutive screening exams and follow every woman for 24 months. Report sensitivity, specificity, recall volume, and how many later-found cancers AI had already flagged.

应用Applications

1. 设备与可穿戴数据：盲法比较商用血氧信号质量或缺氧负荷算法。

   Device and wearable data: compare commercial pulse-oximetry quality or hypoxic-burden algorithms blind.
   在有独立参考标准的片段上本地运行各工具，先把阈值对齐到临床人员的特异度再比较。

   Run each tool locally on segments with an independent reference, and match thresholds to clinician specificity before comparing.
2. 临床文本与语音：把 NLP 过量风险提示当作急诊临床人员旁的「第二阅片者」来评估。

   Clinical text and speech: evaluate an NLP overdose-risk flag as a 'second reader' beside ED clinicians.
   同时报告多发现的病例和多出的警报，并在临床使用前先静默运行。

   Report extra cases found against extra alerts, and run it silently before any clinical use.
3. 罕见事件：借用「富集抽样 + 患病率重加权」，低成本检验可穿戴设备报警。

   Rare events: borrow enriched sampling plus prevalence re-weighting to test wearable alarms cheaply.
   标注全部事件和随机抽取的部分非事件，记录抽样比例，重加权后再报告阳性预测值和报警率。

   Label all events and a random sample of non-events, record the sampling ratio, and re-weight before reporting PPV and alert rates.