关于「缺失非随机:只看到买了的,会把推断带偏多少」,《稀疏购买记录的偏好补全》给出的定量结论是什么?
结论:第五章说明只看到买了的会怎样:选择偏强时未买格子的预测偏高 0.4245,倾向加权降到 0.1671;只拟合已买格子的模型命中率仅 0.286,把没买的格子也计入的加权交替最小二乘达到 0.743。
数据口径:书中顾客数、商品数、口味维度与购买概率均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议,推荐流程只用店铺自有数据,不外传个人数据,不做跨平台追踪。 第七章考辨《周易·系辞上》「方以类聚,物以群分」与斯皮尔曼 1904 年的一般因素,指出纯分群与混合成员下两种模型各有胜负。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.1671, 0.4245
| b = 2, seen 299, true unseen mean 2.6881 | |||||
| method | weight | mean est | unseen bias | unseen RMSE | |
| plain | ess | 299 | 3.7260 | +0.4245 | 0.8633 |
| ipw cap 20 | ess | 158 | 3.1522 | +0.1671 | 0.2619 |
查看完整验算程序
# Chapter 5. p52: inverse-propensity weights against selection bias.
# Model: the store of p51 with selection strength b = 2. When the
# purchase probability pi of each cell is known (in practice it must
# be estimated), weighting each observed cell by 1/pi in ALS and in
# the mean undoes the selection on average. Weights are capped at 20.
# Effective sample size (sum w)^2 / sum w^2 shows the price paid.
# All parameters illustrative.
import math
import random
def dot(a, b):
return math.fsum(x * y for x, y in zip(a, b))
def solve(A, b):
# Gaussian elimination with partial pivoting for a small system
n = len(b)
M = [list(A[i]) + [b[i]] for i in range(n)]
for k in range(n):
p = max(range(k, n), key=lambda i: abs(M[i][k]))
M[k], M[p] = M[p], M[k]
for i in range(k + 1, n):
f = M[i][k] / M[k][k]
M[i] = [M[i][j] - f * M[k][j] for j in range(n + 1)]
x = [0.0] * n
for i in range(n - 1, -1, -1):
s = math.fsum(M[i][j] * x[j] for j in range(i + 1, n))
x[i] = (M[i][n] - s) / M[i][i]
return x
def update(trip, F, r, lam, prior=None):
# weighted ridge step for one row: minimise
# sum w (y - f.x)^2 + lam |x - prior|^2 over x
pr = prior if prior is not None else [0.0] * r
A = [[math.fsum(w * F[j][a] * F[j][b] for j, _, w in trip)
+ (lam if a == b else 0.0) for b in range(r)]
for a in range(r)]
y = [math.fsum(w * F[j][a] * v for j, v, w in trip) + lam * pr[a]
for a in range(r)]
return solve(A, y)
def als(nu, ni, obs, r, lam, iters, seed, V=None, trace=None):
# alternating least squares on (u, i, value, weight) tuples
rng = random.Random(seed)
if V is None:
V = [[rng.gauss(0.0, 1.0) for _ in range(r)] for _ in range(ni)]
bu = [[] for _ in range(nu)]
bi = [[] for _ in range(ni)]
for u, i, x, w in obs:
bu[u].append((i, x, w))
bi[i].append((u, x, w))
U = None
for t in range(iters):
U = [update(bu[u], V, r, lam) for u in range(nu)]
V = [update(bi[i], U, r, lam) for i in range(ni)]
if trace is not None:
trace.append((t + 1, U, V))
return U, V
def factors(n, r, rng):
return [[rng.gauss(0.0, 1.0) for _ in range(r)] for _ in range(n)]
def product(A, B):
return [[dot(a, b) for b in B] for a in A]
def mask(nu, ni, p, rng):
return [(u, i) for u in range(nu) for i in range(ni)
if rng.random() < p]
def logistic(x):
return 1.0 / (1.0 + math.exp(-x))
rng = random.Random(20261113)
nu, ni, r, b = 40, 30, 2, 2.0
F = product([[0.7 * x for x in row] for row in factors(nu, r, rng)],
factors(ni, r, rng))
T = [[3.0 + x for x in row] for row in F]
cells = [(u, i) for u in range(nu) for i in range(ni)]
mu = math.fsum(T[u][i] for u, i in cells) / len(cells)
sd = math.sqrt(math.fsum((T[u][i] - mu) ** 2 for u, i in cells)
/ len(cells))
draws = [rng.random() for _ in cells]
z = [(T[u][i] - mu) / sd for u, i in cells]
lo, hi = -10.0, 10.0
for _ in range(80):
a = (lo + hi) / 2
share = math.fsum(logistic(a + b * x) for x in z) / len(z)
lo, hi = (a, hi) if share < 0.25 else (lo, a)
pi = {c: logistic(a + b * x) for c, x in zip(cells, z)}
seen = [c for c, d in zip(cells, draws) if d < pi[c]]
ss = set(seen)
rest = [c for c in cells if c not in ss]
true = math.fsum(T[u][i] for u, i in rest) / len(rest)
print("b = 2, seen {}, true unseen mean {:.4f}".format(len(seen), true))
print(" method weight mean est unseen bias unseen RMSE")
for name, cap in (("plain", None), ("ipw cap 20", 20.0)):
wt = {c: 1.0 if cap is None else min(cap, 1.0 / pi[c]) for c in seen}
W = math.fsum(wt.values())
m0 = math.fsum(wt[c] * T[c[0]][c[1]] for c in seen) / W
obs = [(u, i, T[u][i] - m0, wt[(u, i)]) for u, i in seen]
U, V = als(nu, ni, obs, r, 0.1, 60, 4)
pr = [m0 + dot(U[u], V[i]) for u, i in rest]
bias = math.fsum(pr) / len(pr) - true
err = math.sqrt(math.fsum((x - T[u][i]) ** 2
for x, (u, i) in zip(pr, rest)) / len(pr))
ess = W * W / math.fsum(x * x for x in wt.values())
print(" {:11s} ess {:4.0f} {:.4f} {:+.4f} {:.4f}".format(
name, ess, m0, bias, err))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《低秩矩阵补全与交替最小二乘、稀疏购买记录推断未表态偏好母典》第 5 章「缺失非随机:只看到买了的,会把推断带偏多少」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-328 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026328,
author = {Fan, Yafu},
title = {Low-Rank Matrix Completion and Alternating Least Squares: Inferring Unstated Preferences from Sparse Purchase Records},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}