关于「抽取:把书稿里能执行的句子一句一句拣出来」,《专著蒸馏与技能包工程》给出的定量结论是什么?

结论:规则句有八种句型:条件—动作、阈值、禁忌、次序、例外、公式、拒绝、定义,标注框是第九种信号。在 30 句的虚构报价手册上,识框的提示词抽取器中文精确率 0.8966、召回率 0.8387,英文精确率 0.8333、召回率 0.9677;标注框信号不花召回率就拿掉了误报,中英并集只漏掉一句不带任何提示词的隐性条件。中文数量词要先换算,换算不了的要拒绝。召回率的结论受金标准条数限制:目标 0.90 而一条不漏,至少要 35 条金标准。两位标注者的 kappa 受规则句比例左右,比较不同书稿时要先看比例。抽取的产物是一张逐栏填满的抽取单,第三章从这张单子出发,把规则句编译成可以执行的谓词。

数据口径:书中样本全部虚构,定价与成交规则只写一般框架,不是法律或合规意见。 全书按证据分级标注每一条说法,以《荀子·劝学》考辨工具与积累,所有计算由书中程序生成,作者的判断一律写进假说框并写明什么结果会推翻它。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.8333, 0.8387, 0.8966, 0.90, 0.9677, 30

recall with 95% Wilson interval
zh overall26/31 = 0.8387 [0.6737, 0.9291]
en overall30/31 = 0.9677 [0.8381, 0.9943]
zh COND3/ 6 = 0.5000 [0.1876, 0.8124]
zh THRESH6/ 7 = 0.8571 [0.4869, 0.9743]
PROHIB3/ 3 = 1.0000 [0.4385, 1.0000]
EXCEPT2/ 2 = 1.0000 [0.3424, 1.0000]
zh recall: micro 0.8387, macro over 8 types 0.8780
en recall: micro 0.9677, macro over 8 types 0.9792
zh: P 0.8966 R 0.8387 F1 0.8667 F2 0.8497 F0.5 0.8844
en: P 0.8333 R 0.9677 F1 0.8955 F2 0.9375 F0.5 0.8571
gold rules needed so that the Wilson lower bound on recall reaches the target
target 0.800 miss -> n 16; 1 miss -> n 25; 2 miss -> n 33; 3 miss -> n 40
target 0.900 miss -> n 35; 1 miss -> n 53; 2 miss -> n 69; 3 miss -> n 84
target 0.950 miss -> n 73; 1 miss -> n 110; 2 miss -> n 142; 3 miss -> n 173
zero misses in n gold rules: one-sided 95% upper bound on miss rate1 - 0.05^(1/n)
n10: exact 0.2589, rule of three 3/n 0.3000
n30: exact 0.0950, rule of three 3/n 0.1000
n 100: exact 0.0295, rule of three 3/n 0.0300
n 300: exact 0.0099, rule of three 3/n 0.0100
查看完整验算程序
# Chapter 2. p21: Wilson intervals for extraction recall, micro vs macro averages, and audit sizes.
# Counts are the box-aware results printed by p20 (TP, FN per type; TP, FP overall).
# Wilson score interval: centre (k + z^2/2) / (n + z^2), half-width z sqrt(k(n-k)/n + z^2/4) / (n + z^2).
# F-beta = (1 + b^2) P R / (b^2 P + R); b = 2 weighs a missed rule four times a false one.
import math
from fractions import Fraction

Z = 1.959964  # two-sided 95 %


def wilson(k, n, z=Z):
    c = (k + z * z / 2) / (n + z * z)
    h = z * math.sqrt(k * (n - k) / n + z * z / 4) / (n + z * z)
    return c - h, c + h


def fbeta(p, r, b):
    return (1 + b * b) * p * r / (b * b * p + r)


# per type (TP, FN) from p20, box-aware
ZH = {"COND": (3, 3), "THRESH": (6, 1), "PROHIB": (3, 0), "ORDER": (4, 0), "EXCEPT": (2, 0),
      "FORMULA": (3, 0), "REFUSE": (3, 0), "DEF": (2, 1)}
EN = {"COND": (5, 1), "THRESH": (7, 0), "PROHIB": (3, 0), "ORDER": (4, 0), "EXCEPT": (2, 0),
      "FORMULA": (3, 0), "REFUSE": (3, 0), "DEF": (3, 0)}
OVERALL = {"zh": (26, 3, 5), "en": (30, 6, 1)}  # TP, FP, FN

if __name__ == "__main__":
    print("   recall with 95% Wilson interval")
    for name, (k, n) in [("zh overall", (26, 31)), ("en overall", (30, 31)), ("zh COND", (3, 6)),
                         ("zh THRESH", (6, 7)), ("PROHIB", (3, 3)), ("EXCEPT", (2, 2))]:
        lo, hi = wilson(k, n)
        print("   {:11s} {:2d}/{:2d} = {:.4f}   [{:.4f}, {:.4f}]".format(name, k, n, k / n, lo, hi))
    for lang, table in (("zh", ZH), ("en", EN)):
        tp = sum(a for a, _ in table.values())
        fn = sum(b for _, b in table.values())
        micro = Fraction(tp, tp + fn)
        macro = sum(Fraction(a, a + b) for a, b in table.values()) / len(table)
        print("   {} recall: micro {:.4f}, macro over 8 types {:.4f}".format(
            lang, float(micro), float(macro)))
    for lang, (tp, fp, fn) in OVERALL.items():
        p, r = tp / (tp + fp), tp / (tp + fn)
        print("   {}: P {:.4f}  R {:.4f}  F1 {:.4f}  F2 {:.4f}  F0.5 {:.4f}".format(
            lang, p, r, fbeta(p, r, 1), fbeta(p, r, 2), fbeta(p, r, 0.5)))
    print("   gold rules needed so that the Wilson lower bound on recall reaches the target")
    for target in (0.80, 0.90, 0.95):
        row = []
        for misses in (0, 1, 2, 3):
            n = misses + 1
            while wilson(n - misses, n)[0] < target:
                n += 1
            row.append("{} miss -> n {}".format(misses, n))
        print("   target {:.2f}: {}".format(target, "; ".join(row)))
    print("   zero misses in n gold rules: one-sided 95% upper bound on miss rate = 1 - 0.05^(1/n)")
    for n in (10, 30, 100, 300):
        print("   n {:3d}: exact {:.4f}, rule of three 3/n {:.4f}".format(n, 1 - 0.05 ** (1 / n), 3 / n))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《专著向智能体技能包无损蒸馏、规则可执行化编译与系统自进化工程》第 2 章「抽取:把书稿里能执行的句子一句一句拣出来」(版本 0.1.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-150 · v0.1.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026150,
  author    = {Fan, Yafu},
  title     = {Lossless Distillation of Monographs into Agent Skill Packs, Executable Rule Compilation and Self-Evolving Systems},
  year      = {2026},
  publisher = {5D Library},
  edition   = {0.1.0},
  note      = {DOI pending}
}

本书其他章节结论