[R-sig-eco] glm-model evaluation

[Ruben Roa Ureta]

> I agree also. The magic alpha < 0.05 is being replaced in literature by an
> equally arbitrary delta > 2, despite all warnings by B&A and others.
>
> As Royall beautifully pointed, there is a key distinction between the
> value of
> data as evidence, and our degree of belief, even after we know this value.
> The
> AIC and log-likelihood ratios gauges the evidence value, while arbitrary
> thresholds are a matter of belief.
>
> The canonical experiment of Royall is a wonderful example: imagine a
> experiment where you take balls at random from an urn, with reposition.
> The
> two competing hypothesis are H1: All balls are black and H2: Half of the
> balls
> are black and half are white. If you take 3 balls and they are all black,
> the
> likelihood ratio in favor of H1 is 8, and thus the log-ratio (and the
> delta-AIC) is 2.07. This number express the comparative evidence value of
> the
> data to one models vis a vis to the other model. If this will *convince*
> you
> that H1 is true is up to you, and not to the statistics. Of course the
> statistical result is important, but it is only part of the general
> argument
> that you frame in the discussion. It is the whole argument, and not only
> the
> evidence value that makes the reader believe in your conclusions.
>
> Paulo

You mean Richard Royall's 1997 book Statistical Evidence: A likelihood
paradigm. Yes, as you point out the canonical experiment also shows that a
delta AIC of 2 is equivalent to a likelihood ratio of 8 (it should be
pointed out that the likelihood ratio we are talking about is the 'pure
likelihood-ratio', not the frequentist test statistic).
I think it is important to have a socialized threshold value, like the
p-value of 0.05, in order to carry out a smooth transition from the
p-value to the likelihood ratio and the delta AIC (or the Akaike weight
ratio, another option). I have tested the canonical experiment with a few
generations of students and they think 8 is not enough, they agree that 16
(4 white balls in a row) is decisive.
Rubén