关于「比特账与词汇空缺度」,《嵌词阈值与单语等效》给出的定量结论是什么?
结论:修正后的比特账把码长与剩余歧义加在一起。在示意参数下,六个空缺词里有两个嵌入后码长更长,可总账上五个更省,只有一个更贵;省下的比特从 +1.33 到 +7.22 不等。空缺度可以用最近邻词的特征失配率来量,但空缺深浅与嵌词收益并不同步,还要看最近邻词周围有多少相似概念。语言各自在简单与精确之间取舍,空缺因此而生;双语者借用另一种语言的分类边界,就能把意思切得更细。
数据口径:嵌词收益只在读者共享双语词库时成立:把单语表达、嵌入与嵌入加注释写成覆盖率的三条成本线,示意参数下 kiasu 在书面渠道的覆盖率阈值为 0.844;读者不发问时,失败会变成沉默的误读;各地区分页面值得嵌的词,合成混合页面后可能全部跌破阈值。 书中参数为示意,全部计算由书中程序生成,不对应任何真实客户、对话或成交;马来语、闽南语、粤语、英语、华语一律平等对待。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 1.33, 7.22
| item | Lm | Le | Hm | He | Tm | Te | Le/Lm | Te/Tm |
|---|---|---|---|---|---|---|---|---|
| kiasu | 19.99 | 14.97 | 2.6 | 0.4 | 22.59 | 15.37 | 0.749 | 0.680 |
| lepak | 14.08 | 14.55 | 2.1 | 0.3 | 16.18 | 14.85 | 1.033 | 0.918 |
| paiseh | 14.95 | 13.55 | 1.8 | 0.5 | 16.75 | 14.05 | 0.906 | 0.839 |
| shiok | 10.44 | 15.14 | 1.5 | 0.6 | 11.94 | 15.74 | 1.450 | 1.318 |
| sajiao | 22.41 | 17.29 | 2.4 | 0.7 | 24.81 | 17.99 | 0.772 | 0.725 |
| yuanfen | 18.08 | 16.29 | 2.9 | 0.9 | 20.98 | 17.19 | 0.901 | 0.819 |
| bits saved for a listener who knows the word (Tm - Te): | ||||||||
| kiasu +7.22 | lepak +1.33 | paiseh +2.70 | ||||||
| shiok -3.80 | sajiao +6.82 | yuanfen +3.80 |
查看完整验算程序
# Chapter 3. p31: code length plus residual ambiguity, six lexical gaps.
# Lm: bits of a monolingual circumlocution under the listener's bigram
# model (product of the listed token probabilities). Le: bits of the
# inserted word, -log2 P(word | shared bilingual lexicon), plus 2 bits of
# switch cue (in and out). Hm, He: residual ambiguity, the expected bits
# still needed to pin the intended sense after each rendering. Every
# probability is an illustrative parameter, not a corpus estimate.
import math
ITEMS = [
# name, circumlocution token probabilities, P(word), Hm, He
("kiasu", [0.05, 0.10, 0.04, 0.08, 0.06], 1 / 8000, 2.6, 0.4),
("lepak", [0.08, 0.12, 0.06, 0.10], 1 / 6000, 2.1, 0.3),
("paiseh", [0.10, 0.07, 0.09, 0.05], 1 / 3000, 1.8, 0.5),
("shiok", [0.12, 0.06, 0.10], 1 / 9000, 1.5, 0.6),
("sajiao", [0.03, 0.05, 0.04, 0.06, 0.05], 1 / 40000, 2.4, 0.7),
("yuanfen", [0.05, 0.04, 0.03, 0.06], 1 / 20000, 2.9, 0.9),
]
CUE = 2.0
def lengths(item):
name, toks, pw, hm, he = item
lm = math.fsum(-math.log2(p) for p in toks)
le = -math.log2(pw) + CUE
return lm, le, hm, he
if __name__ == "__main__":
print("item Lm Le Hm He Tm Te Le/Lm Te/Tm")
for it in ITEMS:
lm, le, hm, he = lengths(it)
tm, te = lm + hm, le + he
print("{:<7} {:>6.2f} {:>6.2f} {:>4.1f} {:>4.1f} {:>6.2f} {:>6.2f}"
" {:>6.3f} {:>6.3f}".format(it[0], lm, le, hm, he, tm, te,
le / lm, te / tm))
gains = []
for it in ITEMS:
lm, le, hm, he = lengths(it)
gains.append((lm + hm) - (le + he))
print("bits saved for a listener who knows the word (Tm - Te):")
print(" " + " ".join("{} {:+.2f}".format(it[0], g)
for it, g in zip(ITEMS[:3], gains[:3])))
print(" " + " ".join("{} {:+.2f}".format(it[0], g)
for it, g in zip(ITEMS[3:], gains[3:])))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《多语参杂语码转换信息论、词汇空缺补偿与单语种等效精准表达母典》第 3 章「比特账与词汇空缺度」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-381 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026381,
author = {Fan, Yafu},
title = {Information Theory of Code-Switching, Lexical-Gap Compensation and Monolingual Precision Equivalence in Multilingual Markets},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}