pscl包hurdle模型response与count预测差异技术问询
问题场景
@Achim Zeileis 以下是我遇到的问题的可复现示例(reprex)。我实际拟合的模型基于约15000个观测值而非100个,但为了便于解释,我对数据做了大幅子集抽样。
在本示例中,count类型预测值小于response类型预测值,我无法从直觉上理解为何会出现这种情况。我95%确定只是自己误解了predict(..., type = 'response')函数的作用,以及它和“用预测的count值乘以未通过hurdle阶段的概率”的计算方式有何区别,希望能得到相关指导。
可复现代码
data <- structure(list(y = c(0, 5, 2, 2, 0, 0, 1, 1, 2, 0, 0, 1, 0, 0, 0, 3, 3, 0, 34, 1, 1, 0, 0, 83, 0, 5, 6, 6, 3, 0, 0, 1, 0, 1, 0, 17, 0, 0, 0, 1, 0, 0, 1, 0, 26, 19, 16, 3, 9, 0, 1, 0, 1, 11, 0, 42, 0, 2, 2, 3, 0, 0, 1, 0, 0, 3, 0, 4, 1, 0, 0, 0, 1, 0, 4, 0, 11, 2, 4, 5, 0, 0, 0, 1, 3, 38, 0, 0, 1, 0, 0, 8, 19, 12, 0, 0, 0, 0, 0, 1), x = c(1, 21, 7, 3, 0, 0, 16, 19, 0, 4, 3, 5, 0, 0, 0, 4, 2, 6, 31, 0, 10, 0, 0, 280, 2, 43, 8, 46, 10, 0, 0, 1, 0, 11, 0, 27, 0, 6, 0, 0, 5, 8, 4, 0, 9, 23, 12, 0, 35, 0, 2, 0, 1, 0, 2, 35, 0, 0, 6, 1, 0, 0, 19, 0, 0, 14, 0, 2, 0, 0, 0, 1, 0, 3, 10, 6, 13, 5, 7, 11, 3, 3, 2, 0, 0, 19, 1, 0, 4, 0, 0, 6, 1, 0, 0, 0, 0, 1, 0, 5)), row.names = c(NA, -100L), class = c("tbl_df", "tbl", "data.frame")) library(pscl) #> Classes and Methods for R developed in the #> Political Science Computational Laboratory #> Department of Political Science #> Stanford University #> Simon Jackman #> hurdle and zeroinfl functions by Achim Zeileis model <- hurdle( y ~ x, data = data, dist = 'negbin' ) newdata <- data.frame(x = 5) # response类型预测值 ( response_pred <- predict(model, newdata = newdata, type = 'response') ) #> 1 #> 2.912046 # count类型预测值 ( count_pred <- predict(model, newdata = newdata, type = 'count') ) #> 1 #> 0.002132521 coef(model) #> count_(Intercept) count_x zero_(Intercept) zero_x #> -6.4765439 0.0652187 -0.9765759 0.3145331 # 手动计算计数部分的指数化线性预测值 ( count_pred_from_coef <- exp(0.0652187 * 5 - 6.4765439) ) #> [1] 0.002132521 # 手动提取零概率预测值 ( predprob_zero <- predict(model, newdata = newdata, type = 'prob')[1] ) #> [1] 0.3552388 # 手动计算逻辑斯蒂回归得到的零概率 ( zero_prob_from_coef <- 1 - (exp(0.3145331 * 5 - 0.9765759) / (1 + exp(0.3145331 * 5 - 0.9765759))) ) #> [1] 0.3552388 # 手动计算的乘积远小于response预测值,是公式错误吗? count_pred * predprob_zero #> 1 #> 0.0007575543
以上代码由reprex包(v1.0.0)于2021-04-06生成
核心原因解释
- 对
pscl包hurdle模型的predict函数各type参数的定义存在误解,是计算结果偏差的核心原因 type = 'count'返回的是计数模型部分(未截断的负二项分布)的均值参数mu,不是非零样本上的条件截断均值type = 'response'返回的是整个栅栏模型的全样本期望,正确的拆解逻辑是:- 栅栏二分类部分给出观测非零的概率
P(y>0) = 1 - P(y=0),示例中该值为1 - 0.355 = 0.645 - 计数部分拟合的是零截断负二项分布,其在非零样本上的条件均值
E[y|y>0],需要用未截断的负二项均值mu除以该负二项分布自身取零值的概率,即mu / P_nb(y=0)。示例中负二项分布的离散参数theta估计值极小,存在重尾特性,尽管mu本身仅为0.00213,经过截断调整后的条件均值可达4.5以上 - 最终全样本期望 =
P(y>0) * E[y|y>0],与type = 'response'的输出2.91完全匹配
- 栅栏二分类部分给出观测非零的概率
- 手动计算时的两处核心错误:
- 误把
count返回的未截断mu当成了截断后的条件均值,即便调整概率相乘的方向也会出现量级偏差 - 直接用mu乘以零概率
P(y=0)而非非零概率,进一步放大了计算结果和真实值的差距
- 误把
内容的提问来源于stack exchange,提问作者Matt Kaye
相关产品推荐
相关产品推荐

