[R-sig-ME] Can AIC be approximated by -2ln(L) i.e. the deviance, at very large sample size?
Chris Howden
chris at trickysolutions.com.au
Mon Mar 4 03:07:27 CET 2013
Thanks Luca,
A quick look at some of my results suggests that that the same ‘large sample
size’ effects carry through to BIC, with it still selecting the full model.
Chris Howden B.Sc. (Hons) GStat.
Founding Partner
Evidence Based Strategic Development, IP Commercialisation and Innovation,
Data Analysis, Modelling and Training
(mobile) 0410 689 945
(fax) +612 4782 9023
chris at trickysolutions.com.au
Disclaimer: The information in this email and any attachments to it are
confidential and may contain legally privileged information. If you are not
the named or intended recipient, please delete this communication and
contact us immediately. Please note you are not authorised to copy, use or
disclose this communication or any attachments without our consent. Although
this email has been checked by anti-virus software, there is a risk that
email messages may be corrupted or infected by viruses or other
interferences. No responsibility is accepted for such interference. Unless
expressly stated, the views of the writer are not those of the company.
Tricky Solutions always does our best to provide accurate forecasts and
analyses based on the data supplied, however it is possible that some
important predictors were not included in the data sent to us. Information
provided by us should not be solely relied upon when making decisions and
clients should use their own judgement.
From: lborger [mailto:lborger at cebc.cnrs.fr]
Sent: Monday, 4 March 2013 11:30 AM
To: Chris Howden; Steve Taylor; Emmanuel Curis; Ben Bolker
Cc: r-sig-mixed-models
Subject: Re: [R-sig-ME] Can AIC be approximated by -2ln(L) i.e. the
deviance, at very large sample size?
Hello,
>I recently did some analysis on "Big Data", the number of rows was over 300
>000. What I found was that the Full Model was always selected using AIC,
>Deviance and LRT. However when I had a look at the effects of the
>predictors I found some of them were negligible, to the point of not really
>being worth including in the model.
You might find this one interesting: Link, W. A., and R. J. Barker. 2006.
Model weights and the foundations of multimodel inference. Ecology
87:2626-2635.
Cheers,
Luca
------------------------------------------------------------------
Luca Borger (PhD, MSc, BMus)
Centre d'Etudes Biologiques de Chize
CNRS (U.P.R. 1934) & INRA (USC 1339)
79360 Villiers-en-Bois, France
*****
email: lborger at cebc.cnrs.fr
Skype: luca.borger | Tel: +33 (0)549 099613
http://cnrs.academia.edu/LucaBorger
http://www.researcherid.com/rid/C-6003-2008
http://www.cebc.cnrs.fr/Fidentite/borger/borger.htm
------------------------------------------------------------------
* new book chapter:
Borger & Fryxell (2012) Quantifying individual differences in dispersal
using the net squared displacement statistics.
Ch. 17 In: Dispersal Ecology and Evolution. Editors: Clobert J., Baguette
M., Benton T., Bullock J.
Oxford University Press, Oxford (UK).
-
-----Original Message-----
From: Chris Howden <chris at trickysolutions.com.au>
To: Steve Taylor <steve.taylor at aut.ac.nz>, Emmanuel Curis
<emmanuel.curis at parisdescartes.fr>, Ben Bolker <bbolker at gmail.com>
Cc: r-sig-mixed-models <r-sig-mixed-models at r-project.org>
Date: Mon, 4 Mar 2013 11:04:01 +1100
Subject: Re: [R-sig-ME] Can AIC be approximated by -2ln(L) i.e. the
deviance, at very large sample size?
Thanks for the responses everyone,
I agree that its changes of 'goodness of fit' Likelihood functions such as
AIC and deviance that matter, not their absolute size.
However I think the impact of sample size may be something we need to
consider, particularly when analysing "Big Data" sets.
I recently did some analysis on "Big Data", the number of rows was over
300 000. What I found was that the Full Model was always selected using
AIC, Deviance and LRT. However when I had a look at the effects of the
predictors I found some of them were negligible, to the point of not
really being worth including in the model. Despite what the AIC and LRT
say.
This seems to be the same sample size issue faced with simple Univariate
tests such as ANOVA i.e. large sample sizes give so much power that
statistically significant results may be of no/little practical value.
The reason I asked about the convergence of deviance and AIC at large
sample sizes was thus.
The LRT tests between the Full model and 1 less predictor all had
exceptionally small p-values, which meant that the difference in Ln(L) was
very large. So large that it appears that the difference in deviance and
AIC was essentially the same.
So although it’s the difference that matters, if they converge at large
sample sizes than a large difference in deviance means there will also be
a large difference in AIC and they will come to the same conclusion??
However as they don't converge at small sample sizes this effect is not as
relevant.
Chris Howden B.Sc. (Hons) GStat.
Founding Partner
Evidence Based Strategic Development, IP Commercialisation and Innovation,
Data Analysis, Modelling and Training
(mobile) 0410 689 945
(fax) +612 4782 9023
chris at trickysolutions.com.au
Disclaimer: The information in this email and any attachments to it are
confidential and may contain legally privileged information. If you are
not the named or intended recipient, please delete this communication and
contact us immediately. Please note you are not authorised to copy, use or
disclose this communication or any attachments without our consent.
Although this email has been checked by anti-virus software, there is a
risk that email messages may be corrupted or infected by viruses or other
interferences. No responsibility is accepted for such interference. Unless
expressly stated, the views of the writer are not those of the company.
Tricky Solutions always does our best to provide accurate forecasts and
analyses based on the data supplied, however it is possible that some
important predictors were not included in the data sent to us. Information
provided by us should not be solely relied upon when making decisions and
clients should use their own judgement.
-----Original Message-----
From: Steve Taylor [mailto:steve.taylor at aut.ac.nz]
Sent: Monday, 4 March 2013 10:10 AM
To: Emmanuel Curis; Chris Howden
Cc: r-sig-mixed-models
Subject: RE: [R-sig-ME] Can AIC be approximated by -2ln(L) i.e. the
deviance, at very large sample size?
I agree that it is changes in AIC that matter, not its absolute value.
My understanding is that AIC is only useful for comparing two models
fitted on the same data set, i.e. with the same sample size. So the
question of how AIC changes with sample size is of little use beyond
curiosity.
The change in AIC caused by adding a term to the model formula would be of
interest. But the change in AIC caused by adding cases to the sample size
is pretty meaningless.
The 2K part is important because it provides a penalty for the change in
the number of parameters between a simpler model and a more complex model.
I would advise against making any approximations when calculating AIC,
especially considering its main use is in taking the difference between
two close large numbers.
cheers,
Steve
-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org
[mailto:r-sig-mixed-models-bounces at r-project.org] On Behalf Of Emmanuel
Curis
Sent: Friday, 1 March 2013 9:18p
To: Chris Howden
Cc: r-sig-mixed-models
Subject: Re: [R-sig-ME] Can AIC be approximated by -2ln(L) i.e. the
deviance, at very large sample size?
Hi,
I may be wrong, but I understood that AIC in itself is not as important as
changes in AIC between models, and some authors says that changes in AIC
in the order of more than 10 are enough to favor a model on another.
And changes in the 2*k term should be in this order of magnitude when
comparing different models.
So my guess would be that it remains important.
On the other hand, if a set of parameters will remain in all models, it
probably can be safely ignored in the 2*k term for all models.
Hope this helps,
On Fri, Mar 01, 2013 at 06:30:53PM +1100, Chris Howden wrote:
< Hi everyone,
<
< Although not strictly an R issue there often seems to be discussions
along < these lines on this list, so I hope no one minds me posting this.
If U do < please let me know. (and just for the record I am applying this
in R) < < I'm trying to get my head around AIC and sample size.
<
< Now if AIC = -2ln(L) + 2K = Deviance + 2K < < Am I right in thinking
that as the Likelihood is the product of < probabilities then (all else
being equal) the larger the sample size the < smaller the Likelihood?
< Which means that if we have very large sample sizes we expect the
-2ln(L) < term to be a very large number?
< Which would reduce the effect of the parameter correction term 2K?
<
<
< Chris Howden B.Sc. (Hons) GStat.
< Founding Partner
< Evidence Based Strategic Development, IP Commercialisation and
Innovation, < Data Analysis, Modelling and Training < (mobile) 0410 689
945 < (fax) +612 4782 9023 < chris at trickysolutions.com.au < <
_______________________________________________
< R-sig-mixed-models at r-project.org mailing list <
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
--
Emmanuel CURIS
emmanuel.curis at parisdescartes.fr
Page WWW: http://emmanuel.curis.online.fr/index.html
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
More information about the R-sig-mixed-models
mailing list