[R-sig-ME] Comparing Model Performance Across Data Sets: report p values?
Karista Hudelson
karistaeh at gmail.com
Tue Aug 8 18:38:20 CEST 2017
Hello again List,
Thanks for the clarification question Thierry. I want to compare the
predictive ability of the model terms between the two phases. For
instance, is Sea Ice more important in phase 1? This comparison is
confounded somewhat by the unequal sample sizes in the phases I think, but
am not sure. Maybe that is part of my question: should I focus less on the
p values (as Phillip recommends in his first point I think) and instead
look at the overall model fit for each phase?
Phillip, thank you for your second suggestion! I followed your advice and
included Phase in the model and also tried running it with interactions
between the fixed effects and phase.
*Here is the model without phase:*
FSVlmer1a<-lmer(logHg~Length+Res_Sea_Ice_Dur+Spring_MST+Summer_Rain+(1|WA),data=FSV2)
REML criterion at convergence: -389.3
Scaled residuals:
Min 1Q Median 3Q Max
-6.1650 -0.6235 -0.0447 0.6380 3.0889
Random effects:
Groups Name Variance Std.Dev.
WA (Intercept) 0.11493 0.3390
Residual 0.03244 0.1801
Number of obs: 790, groups: WA, 5
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) -1.064e+00 1.971e-01 1.130e+01 -5.399 0.000195 ***
Length 2.204e-02 1.105e-03 7.817e+02 19.952 < 2e-16 ***
Res_Sea_Ice_Dur 7.917e-04 2.977e-04 7.813e+02 2.660 0.007978 **
Spring_MST 1.892e-02 4.514e-03 7.812e+02 4.190 3.11e-05 ***
Summer_Rain -2.194e-03 3.650e-04 7.811e+02 -6.011 2.82e-09 ***
---
> sem.model.fits(FSVlmer1a)
Class Family Link n Marginal Conditional
1 merModLmerTest gaussian identity 790 0.127793 0.8080115
> AIC(FSVlmer1a)
[1] -375.2507
*Same model with Phase interactions:*
>
FSV2lmer1bi<-lmer(logHg~Length*Phase+Res_Sea_Ice_Dur*Phase+Spring_MST*Phase+Summer_Rain*Phase+(1|WA),data=FSV2)
REML criterion at convergence: -360.9
Scaled residuals:
Min 1Q Median 3Q Max
-6.2490 -0.6285 -0.0176 0.6076 3.1211
Random effects:
Groups Name Variance Std.Dev.
WA (Intercept) 0.11988 0.3462
Residual 0.03195 0.1788
Number of obs: 790, groups: WA, 5
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) -1.179e+00 2.122e-01 1.400e+01 -5.556 7.10e-05
***
Length 2.146e-02 1.204e-03 7.767e+02 17.827 < 2e-16
***
*Phasepre 8.858e-01 3.945e-01 7.765e+02 2.246 0.025014
* *
Res_Sea_Ice_Dur 1.389e-03 3.963e-04 7.763e+02 3.504 0.000484
***
Spring_MST 1.680e-02 4.924e-03 7.761e+02 3.411 0.000681
***
Summer_Rain -2.254e-03 3.980e-04 7.760e+02 -5.664 2.08e-08
***
*Length:Phasepre 2.582e-03 2.917e-03 7.762e+02 0.885 0.376294
*
*Phasepre:Res_Sea_Ice_Dur -4.806e-03 1.607e-03 7.764e+02 -2.990 0.002876
** *
*Phasepre:Spring_MST -8.681e-03 2.147e-02 7.760e+02 -0.404 0.686088
*
*Phasepre:Summer_Rain -4.634e-03 2.072e-03 7.764e+02 -2.236 0.025636
* *
> AIC(FSV2lmer1bi)
[1] -336.8567
> sem.model.fits(FSV2lmer1bi,aicc=T)
Class Family Link n Marginal Conditional
1 merModLmerTest gaussian identity 790 0.126233 0.8161111
So the overall fit metrics for these two models are not so different, and
the simpler one is a bit better.
And in case it would be helpful/interesting, here are the fits of the
models for phase 1 and phase 2 (which were described in my first question):
FSV2lmer1apre<-lmer(logHg~Length+Res_Sea_Ice_Dur+Spring_MST+Summer_Rain+(1|WA),data=FSV2pre)
# AIC 10.06269, R2s:0.1508716 0.7681201
FSV2lmer1apost<-lmer(logHg~Length+Res_Sea_Ice_Dur+Spring_MST+Summer_Rain+(1|WA),data=FSV2post)
# AIC -335.1748, R2s: 0.1233518 0.8228584
Thank you Phillip and Thierry for your kind and encouraging attention to
this question. I hope I can trouble you and the rest of the list for a bit
more instruction on this/these questions, as this issue is the crux of the
interpretation of this data.
Looking forward to your thoughts and suggestions,
Karista
On Thu, Aug 3, 2017 at 4:40 AM, Phillip Alday <phillip.alday at mpi.nl> wrote:
> Dear Karista,
>
> as Thierry said, knowing more about the inferences you want to make will
> get you better advice here. That said, I do have two suggestions in the
> meantime:
>
> 1. Don't focus on significance, especially of individual predictors, as
> much as estimates and overall model fit / predictive ability. (cf. The
> New Statistics, The Difference between Significant and Insignificant is
> not itself Significant, Choosing prediction over explanation in
> psychology, etc.)
>
> 2. Put all your data into one model and include time period as a fixed
> effect. Such pooling will generally help all your estimates; moreover,
> it gives you a more principled way to compare time periods (both in the
> main effect of time period and in its interactions with individual
> variables).
>
> Best,
> Phillip
>
> On 08/03/2017 10:20 AM, Thierry Onkelinx wrote:
> > Dear Karista,
> >
> > Much depends on what you want to compare between the models. The
> parameter
> > estimates? The predicted values? The goodness of fit? You 'll need to
> make
> > that clear.
> >
> > Best regards,
> >
> >
> > ir. Thierry Onkelinx
> > Instituut voor natuur- en bosonderzoek / Research Institute for Nature
> and
> > Forest
> > team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance
> > Kliniekstraat 25
> > 1070 Anderlecht
> > Belgium
> >
> > To call in the statistician after the experiment is done may be no more
> > than asking him to perform a post-mortem examination: he may be able to
> say
> > what the experiment died of. ~ Sir Ronald Aylmer Fisher
> > The plural of anecdote is not data. ~ Roger Brinner
> > The combination of some data and an aching desire for an answer does not
> > ensure that a reasonable answer can be extracted from a given body of
> data.
> > ~ John Tukey
> >
> > 2017-08-02 19:54 GMT+02:00 Karista Hudelson <karistaeh at gmail.com>:
> >
> >> Hello All,
> >>
> >> I am comparing the fit of a mixed model on different time periods of a
> data
> >> set. For the first time period I have 113 observations and only one of
> the
> >> fixed effects is significant. For the second time period I have 322
> >> observations and all of the fixed effects are significant. Because n is
> >> important in the calculation of p, I'm not sure how or even if to
> interpret
> >> the differences in p values for the model terms in the two time periods.
> >> Does anyone have advice on how to compare the fit of the variables in
> the
> >> mixed model for the two data sets in a way that is less impacted by the
> >> difference in the number of observations? Or is a difference of 209
> >> observations enough to drive these differences in p values?
> >>
> >> Time period 1 output:
> >> Fixed effects:
> >> Estimate Std. Error df t value Pr(>|t|)
> >> (Intercept) -0.354795 0.811871 82.140000 -0.437 0.663
> >> Length 0.024371 0.003536 106.650000 6.892 4.01e-10 ***
> >> Res_Sea_Ice_Dur -0.002408 0.002623 107.970000 -0.918 0.361
> >> Sp_MST 0.014259 0.024197 106.310000 0.589 0.557
> >> Summer_Rain -0.005015 0.003536 107.970000 -1.418 0.159
> >>
> >>
> >> Time period 2 output:
> >> Fixed effects:
> >> Estimate Std. Error df t value Pr(>|t|)
> >> (Intercept) -1.183e+00 3.103e-01 6.650e+00 -3.812 0.007281 **
> >> Length 1.804e-02 1.623e-03 3.151e+02 11.120 < 2e-16 ***
> >> Res_Sea_Ice_Dur 2.206e-03 5.929e-04 3.153e+02 3.721 0.000235 ***
> >> Spring_MST 1.022e-02 7.277e-03 3.150e+02 1.404 0.161319
> >> Summer_Rain -1.853e-03 5.544e-04 3.150e+02 -3.343 0.000929 ***
> >>
> >>
> >>
> >>
> >> Thanks in advance for your time and consideration of this question.
> >> Karista
> >>
> >> [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-mixed-models at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >>
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>
--
Karista
[[alternative HTML version deleted]]
More information about the R-sig-mixed-models
mailing list