[R-sig-phylo] Model-Selection vs. Finding Models that "Fit Well"
dwbapst at uchicago.edu
Thu Jan 20 18:27:37 CET 2011
I'd like to pose a question to this group, as a bit of topical
discussion. I apologize in advance if I should mangle a concept.
In many model-based PCMs and some other analyses (such as paleoTS), we
fit models to data by finding the ML estimates of the parameters
associated, calculate the maximum support of each model and than
compare between models with differing parameters using an information
criterion (AICc being probably the most used). Akaike weights can be
calculated if we want to consider the relative fit between our models.
This is contrary to traditional statistics, where alternative
hypotheses are tested against some null hypothesis. Obviously, the
later approach has proven to be thorny because rejecting some null
hypotheses are very difficult (such as a random walk) and some
situations truly lack a clear null model.
Recently, I have heard the opinion expressed from workers of disparate
fields (philosophy, ecology, etc.) that model-choosing methods may
choose the best model, but with no idea of whether any of the models
considered "fit well" to the data or not. In other words, we may have
fit models A-D, and the best model may have been model C, but none of
the models compared could describe the 'true' process underlying the
data at all.
This view gives me mixed feelings. Certainly, if we are using a
model-selection approach, we should attempt the range of models that
make sense for our data, and should particularly include that set of
simplistic models that we may accept as the most observed process
(Brownian motion and Ornstein-Uhlenbeck with one optima, perhaps, in
analyses of trait evolution). Of course, we cannot include models that
we haven't even considered or are analytically intractable. That's a
fundamental limitation of science, however, not model-selection based
This counter-argument did not seem to satisfy the others, who still
wanted a measure of absolute fit, "like an R-squared". Now, perhaps
I'm confused, but isn't R-squared technically a relative measure of
fit between a linear model and a random scatter? I suppose the maximum
support for a model is a measure of absolute fit, but it's not useful
or interpretable unless I'm comparing it to the support for some other
So, it seems like the desire for a measure of absolute fit is not
well-founded, but maybe I'm wrong. Is there something more we can do
to show how that the models we've picked aren't arbitrary? Opinions?
-Dave Bapst, UChicago Geosci
More information about the R-sig-phylo