[R-sig-ME] Confidence interval around random effect variances in place of p-value
Ben Bolker
bbo|ker @end|ng |rom gm@||@com
Mon Apr 5 00:58:10 CEST 2021
This would make an interesting simulation and/or theoretical exercise
(I'm going to resist the urge to do it), i.e. identifying the
correspondence between p-values constructed from parametric bootstrap
full-vs-reduced model comparisons and p-values estimated as fraction of
PB fits of full model that give variance=0 for the tested variance
component(s).
On 4/2/21 8:22 PM, Jack Solomon wrote:
> Well, how about concluding so:
>
> If a (say 2-level) model gives a singular fit (even though perhaps there
> is a "tol" that is small but not exactly "0" for that warning to show
> up), that would mean we have a "practically" non-significant
> random-effect variance component.
>
>
>
> On Fri, Apr 2, 2021 at 7:15 PM Ben Bolker <bbolker using gmail.com
> <mailto:bbolker using gmail.com>> wrote:
>
> I'm not sure that the bootstrapped CIs *wouldn't* work; they might
> return the correct proportion of singular fits ...
>
> On 4/2/21 8:12 PM, Jack Solomon wrote:
> > Thank you all very much. So, I can conclude that a likelihood
> ratio test
> > and/or a parametric bootstrapping can be used for random effect
> variance
> > component hypothesis testing.
> >
> > But I also concluded that the idea of simply using a bootstrapped
> CI for
> > a random-effect variance component [e.g., in lme4;
> > confint(model,method="boot",oldNames=FALSE) ] by definition
> can't be
> > used for significance testing, because it requires the
> possibility of
> > seeing sd = 0 which can't be "strictly" captured by such a CI from a
> > multilevel model (at least not easily so).
> >
> > I hope my conclusions are correct,
> > Thank you all, Jack
> >
> > On Fri, Apr 2, 2021 at 6:51 PM Ben Bolker <bbolker using gmail.com
> <mailto:bbolker using gmail.com>
> > <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>> wrote:
> >
> > Sure. If all you want is p-values, I'd recommend parametric
> > bootstrapping (implemented in the pbkrtest package) ... that
> will avoid
> > these difficulties. (I would also make sure that you know
> *why* you
> > want p-values on the random effects ... they have all of the
> issues of
> > regular p-values plus some extras:
> >
> http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects
> <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>
> >
> <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>>
> >
> > )
> >
> > On 4/2/21 7:37 PM, Jack Solomon wrote:
> > > Thanks. Just to make sure, to declare a statistically
> > NON-significant
> > > random effect variance component, the lower bound of the
> CI must be
> > > EXACTLY "0", right?
> > >
> > > Tha is, for example, a CI like: [.0002, .14] is a
> > > statistically significant random-effect variance component but
> > one that
> > > perhaps borders a p-value of relatively close to but
> smaller than
> > .05,
> > > right?
> > >
> > > On Fri, Apr 2, 2021 at 6:19 PM Ben Bolker
> <bbolker using gmail.com <mailto:bbolker using gmail.com>
> > <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>
> > > <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>
> <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>>> wrote:
> > >
> > > This seems like a potential can of worms (as
> indeed are all
> > > hypothesis tests of null values on a boundary ...)
> However,
> > in this
> > > case
> > > bootstrapping (provided you have resampled appropriately -
> > you may need
> > > to do hierarchical bootstrapping ...) seems reasonable,
> > because a null
> > > model would give you singular fits (i.e. estimated
> sd=0) half
> > of the
> > > time ...
> > >
> > > Happy to hear more informed opinions.
> > >
> > > On 4/2/21 6:55 PM, Jack Solomon wrote:
> > > > Dear All,
> > > >
> > > > A colleague of mine suggested that I use the
> bootstrapped CIs
> > > around my
> > > > model's random effect variances in place of
> p-values for them.
> > > >
> > > > But random effect variances (or sds) start from
> "0". So,
> > to declare a
> > > > statistically NON-significant random effect variance
> > component, the
> > > > lower bound of the CI must be EXACTLY "0", right?
> > > >
> > > > Thank you very much,
> > > > Jack
> > > >
> > > > [[alternative HTML version deleted]]
> > > >
> > > > _______________________________________________
> > > > R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>
> > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>>
> > > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>
> > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>>> mailing list
> > > >
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> > <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
> > >
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> > <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>>
> > > >
> > >
> > > _______________________________________________
> > > R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>
> > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>>
> > > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>
> > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>>> mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> > <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
> > >
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> > <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>>
> > >
> >
>
More information about the R-sig-mixed-models
mailing list