[R-sig-eco] glm-model evaluation

[Paulo Inácio de Knegt López de Prado]

I agree also. The magic alpha < 0.05 is being replaced in literature by an
equally arbitrary delta > 2, despite all warnings by B&A and others. 

As Royall beautifully pointed, there is a key distinction between the value of
data as evidence, and our degree of belief, even after we know this value. The
AIC and log-likelihood ratios gauges the evidence value, while arbitrary
thresholds are a matter of belief.

The canonical experiment of Royall is a wonderful example: imagine a
experiment where you take balls at random from an urn, with reposition. The
two competing hypothesis are H1: All balls are black and H2: Half of the balls
are black and half are white. If you take 3 balls and they are all black, the
likelihood ratio in favor of H1 is 8, and thus the log-ratio (and the
delta-AIC) is 2.07. This number express the comparative evidence value of the
data to one models vis a vis to the other model. If this will *convince* you
that H1 is true is up to you, and not to the statistics. Of course the
statistical result is important, but it is only part of the general argument
that you frame in the discussion. It is the whole argument, and not only the
evidence value that makes the reader believe in your conclusions. 

Paulo
----------------------------------------------------------------------------
> I agree with Ben.  The problem with using a threshold such as the 2
> delta unit rule is that there is a tendency to draw a line in the sand
> and ignore everything on one side.  It should be remembered that
> models with greater than 2 dAIC scores still have SOME explanatory
> power, and that the delta units are arbitrary.  This is why we
> calculate weights, and why B & A advocate so strongly for multimodel
> inference.

> BriAnne

2008/5/29 Ben Bolker <bolker at ufl.edu>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ruben Roa Ureta wrote:
> |> -----BEGIN PGP SIGNED MESSAGE-----
> | [snip]
> |
> |> ~  not absolutely sure what your question is.
> |>
> |> ~  If you're talking about evaluating the relative merit of
> |> the selected model, it's a question of delta-AIC (or delta-AICc),
> |> follow the usual rules of thumb -- <2 is approximately equivalent,
> |> |6 is a lot better, >10 is so good that you can probably discard
> |> worse models.  (See Shane Richards' nice papers on the topic.)
> |
> | I have traced the rule about 2 as the minimum difference to favour one
> | model over the other to remark 2, Ch. 4, Sakamoto, Ishiguro and Kitagawa,
> | 1986, Akaike Information Criterion Statistics. D. Reídle Publishing Co,
> | Dordrecht. They use the expression 'significant difference between
> | models'. However, they do not explain why they think that 2 is the minimum
> | 'significant' delta AIC. Does anybody know more about a justification for
> | this threshold?
> | Rubén
>
> ~  I would really strongly recommend AGAINST trying to justify
> "significance thresholds" for AIC (B&A 2002 say this too).
> - -2 AIC points corresponds to adding a single parameter with no
> explanatory power at all, so it makes sense to me to consider
> this a "minimal change" in the penalized GOF/expected K-L
> distance/whatever.  You can also consider this in terms
> of AIC weights, which in the limit of large sample sizes and
> particular (slightly odd) Bayesian priors have an interpretation
> in terms of posterior model probabilities.
>
> ~  Anyone else?
>
> ~  Ben Bolker
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFIPvQzc5UpGjwzenMRAqG8AJ9VFJtuogFhuDGzMvqsWzSCx4KGxQCfZ8+9
> JxfaMMJI2XtamnDULo5z4DE=
> =khBX
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>



-- 
~~~~~~~~~~~~~~~~~~~~~~~~~
BriAnne Addison
Ecology Evolution & Systematics
University of Missouri - St Louis

--
Paulo Inácio de Knegt López de Prado
Depto. de Ecologia - Instituto de Biociências - USP
Rua do Matão, travessa 14, nº 321
Cid. Universitária, São Paulo - SP
CEP 05508-900
11-30917599 (sala)
11-30917600 (Secretaria)