[R] Zero-inflated regression models: predicting no 0s

[Achim Zeileis]

On Tue, 31 May 2011, Jean-Simon Michaud wrote:

> Hi all,
>
> First post for me here, but I have been reading on the forum for almost two
> years now. Thanks to everyone who contributed btw!
>
> I have a dataset of 4000 observations of count of a mammal and I am trying
> to predict abundance from a inflated-zero model as there is quite a bit of
> zeros in the response variable.
>
> I have tried multiple options, but I might do something wrong as every 
> time I look at the fitted values it do not comprise any 0.

Yes, the fitted() method and also the default of the predict() method is 
to compute fitted _means_. And the mean of a count distribution will 
always be non-zero.

Moreover, even when you round the fitted values to integers, this does 
_not_ lead to the most likely count. Consider the following simple 
examples:

Probability density for a Poisson distribution with mean 0.8

R> dpois(0:5, lambda = 0.8)
[1] 0.449329 0.359463 0.143785 0.038343 0.007669 0.001227

i.e., zero is still the most likely outcome with a probability of 45% even 
though the mean is 0.8. And for negative binomial distributions, this can 
be even more extreme. The probability density for a geometric distribution 
(negative binomial with size = 1) and mean 2:

R> dnbinom(0:5, mu = 2, size = 1)
[1] 0.33333 0.22222 0.14815 0.09877 0.06584 0.04390

i.e., despite the mean of 2, zero is still the most likely outcome.

You can get the predicted probabilities for all observations via
predict(hurdle1, type = "prob") and predict(zip1A, type = "prob"), 
respectively. Given the results for your negative binomial hurdle model, I 
suspect that the zero-inflated Poisson fit will have a nonsatisfactory fit 
for the zeros, even though the predicted means of the two models are 
similar.

See also the paper accompanying the count regression functions in "pscl" 
(not "lpsc"):

   http://www.jstatsoft.org/v27/i08/

Finally, a short comment on your model formula:

   hurdle(TOT ~ LC80 + LC231 + DEM, data = mydata_purge, ...)

will be easier to read and less confusing (after all "data = food" does 
not appear to be used at all).

hth,
Z

> Here is what I tried so far:
>
> "
> ## - hurdle from the package (lpsc) - ##
>
>> hurdle1 = hurdle(formula = mydata_purge2$TOT ~ mydata_purge2$LC80 +
> mydata_purge2$LC231 + mydata_purge2$DEM, data = food, dist = "negbin",
> zero.dist = "binomial")
>
>> summary(hurdle1)
>
> Call:
> hurdle(formula = mydata_purge2$TOT ~ mydata_purge2$LC80 +
> mydata_purge2$LC231 + mydata_purge2$DEM, data = food,
>    dist = "negbin", zero.dist = "binomial")
>
> Pearson residuals:
>    Min      1Q  Median      3Q     Max
> -1.0833 -0.7448 -0.2801  0.4296  6.7242
>
> Count model coefficients (truncated negbin with log link):
>                      Estimate Std. Error z value Pr(>|z|)
> (Intercept)          1.7841678  0.0923781  19.314  < 2e-16 ***
> mydata_purge2$LC80  -2.5929984  0.4184956  -6.196 5.79e-10 ***
> mydata_purge2$LC231  0.2154269  0.1171259   1.839 0.065875 .
> mydata_purge2$DEM    0.0007708  0.0002064   3.735 0.000188 ***
> Log(theta)           0.3742602  0.0390319   9.589  < 2e-16 ***
>
> Zero hurdle model coefficients (binomial with logit link):
>                      Estimate Std. Error z value Pr(>|z|)
> (Intercept)          0.0602347  0.2302370   0.262 0.793614
> mydata_purge2$LC80  -3.0590108  0.8360020  -3.659 0.000253 ***
> mydata_purge2$LC231  1.7754441  0.3226731   5.502 3.75e-08 ***
> mydata_purge2$DEM    0.0031943  0.0005307   6.020 1.75e-09 ***
> ---
> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
>
> Theta: count = 1.4539
> Number of iterations in BFGS optimization: 12
> Log-likelihood: -1.251e+04 on 9 Df
>
>
> ## - zeroinfl from the package (lpsc) - ##
>
>> zip1A = zeroinfl(mydata_purge2$TOT ~ mydata_purge2$LC80 +
> mydata_purge2$LC231 + mydata_purge2$DEM, data = food)
>
>> summary(zip1A)
>
> Call:
> zeroinfl(formula = mydata_purge2$TOT ~ mydata_purge2$LC80 +
> mydata_purge2$LC231 + mydata_purge2$DEM, data = food)
>
> Pearson residuals:
>    Min      1Q  Median      3Q     Max
> -2.2128 -1.2886 -0.5010  0.7594 11.8458
>
> Count model coefficients (poisson with log link):
>                      Estimate Std. Error z value Pr(>|z|)
> (Intercept)          1.894e+00  3.547e-02  53.401  < 2e-16 ***
> mydata_purge2$LC80  -2.249e+00  1.768e-01 -12.725  < 2e-16 ***
> mydata_purge2$LC231  1.799e-01  4.492e-02   4.005 6.21e-05 ***
> mydata_purge2$DEM    6.670e-04  7.687e-05   8.678  < 2e-16 ***
>
> Zero-inflation model coefficients (binomial with logit link):
>                      Estimate Std. Error z value Pr(>|z|)
> (Intercept)         -0.0593751  0.2308068  -0.257 0.796986
> mydata_purge2$LC80   2.9428092  0.8523669   3.453 0.000555 ***
> mydata_purge2$LC231 -1.7772101  0.3233166  -5.497 3.87e-08 ***
> mydata_purge2$DEM   -0.0031901  0.0005319  -5.997 2.01e-09 ***
> ---
> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
>
> Number of iterations in BFGS optimization: 13
> Log-likelihood: -1.727e+04 on 8 Df
>
>> a1 = predict(zip1A)
>> b1 = mydata_purge2$TOT
>> plot(a1,b1)
>
> "
>
> Please find attached the plot of zip1A (which look quite similar to the
> hurdle1).
>
> Your help would be much appreciated,
>
> Thanks,
>
> JM.