[R] Comparison: glm() vs. bigglm()
bcarvalh at jhsph.edu
Fri Jun 29 17:29:42 CEST 2007
thank you very much for your feedback.
As for your observations, I do realize that I'm using 1.5 chunks for
this particular case (10e6 gives around 8 chunks on other sets).
I just noticed that I didn't add the difference in the deviances that
Thank you very much for the suggestion, I'll give it a try.
On Jun 29, 2007, at 11:05 AM, Peter Dalgaard wrote:
> Benilton Carvalho wrote:
>> Until now, I thought that the results of glm() and bigglm() would
>> coincide. Probably a naive assumption?
>> Anyways, I've been using bigglm() on some datasets I have available.
>> One of the sets has >15M observations.
>> I have 3 continuous predictors (A, B, C) and a binary outcome (Y).
>> And tried the following:
>> m1 <- bigglm(Y~A+B+C, family=binomial(), data=dataset1,
>> m2 <- bigglm(Y~A*B+C, family=binomial(), data=dataset1,
>> imp <- m1$deviance-m2$deviance
>> For my surprise "imp" was negative.
>> I then tried the same models, using glm() instead... and as I
>> expected, "imp" was positive.
>> I also noticed differences on the coefficients estimated by glm() and
>> bigglm() - small differences, though, and CIs for the coefficients (a
>> given coefficient compared across methods) overlap.
>> Are such incrongruences expected? What can I use to check for
>> convergence with bigglm(), as this might be one plausible cause for a
>> negative difference on the deviances?
> It doesn't sound right, but I cannot reproduce your problem on a
> sized problem (it pretty much killed my machine...). Some
> A: You do realize that you are only using 1.5 chunks? (15M vs. 10e6
> B: Deviance changes are O(1) under the null hypothesis but the
> themselves are O(N). In a smaller variant (N=1e5), I got
>  138626.4
>  138626.4
>> m2$deviance - m1$deviance
>  -0.05865785
> This does leave some scope for roundoff to creep in. You may want to
> play with a lower setting of tol=...
> O__ ---- Peter Dalgaard Øster Farimagsgade 5, Entr.B
> c/ /'_ --- Dept. of Biostatistics PO Box 2099, 1014 Cph. K
> (*) \(*) -- University of Copenhagen Denmark Ph: (+45)
> ~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk) FAX: (+45)
More information about the R-help