[R-sig-ME] Confidence interval around random effect variances in place of p-value

Mon Apr 5 02:18:00 CEST 2021

Surely any correspondence would be attenuated - IIRC there’s some tuned shrinkage towards zero in the estimates of the variance components?

Best wishes,

Andrew

--
Andrew Robinson
Director, CEBRA, and Professor of Biosecurity Risk and Applied Statistics
Schools of BioSciences and Mathematics & Statistics
University of Melbourne, VIC 3010 Australia
Tel: (+61) 0403 138 955
Email: apro using unimelb.edu.au
Website: http://cebra.unimelb.edu.au/
On 5 Apr 2021, 8:58 AM +1000, Ben Bolker <bbolker using gmail.com>, wrote:
This would make an interesting simulation and/or theoretical exercise
(I'm going to resist the urge to do it), i.e. identifying the
correspondence between p-values constructed from parametric bootstrap
full-vs-reduced model comparisons and p-values estimated as fraction of
PB fits of full model that give variance=0 for the tested variance
component(s).

On 4/2/21 8:22 PM, Jack Solomon wrote:
Well, how about concluding so:

If a (say 2-level) model gives a singular fit (even though perhaps there
is a "tol" that is small but not exactly "0" for that warning to show
up), that would mean we have a "practically" non-significant
random-effect variance component.

On Fri, Apr 2, 2021 at 7:15 PM Ben Bolker <bbolker using gmail.com
<mailto:bbolker using gmail.com>> wrote:

    I'm not sure that the bootstrapped CIs *wouldn't* work; they might
return the correct proportion of singular fits ...

On 4/2/21 8:12 PM, Jack Solomon wrote:
Thank you all very much. So, I can conclude that a likelihood
ratio test
and/or a parametric bootstrapping can be used for random effect
variance
component hypothesis testing.

But I also concluded that the idea of simply using a bootstrapped
CI for
a random-effect variance component [e.g., in lme4;
confint(model,method="boot",oldNames=FALSE)  ] by definition
can't be
used for significance testing, because it requires the
possibility of
seeing sd = 0 which can't be "strictly" captured by such a CI from a
multilevel model (at least not easily so).

I hope my conclusions are correct,
Thank you all, Jack

On Fri, Apr 2, 2021 at 6:51 PM Ben Bolker <bbolker using gmail.com
<mailto:bbolker using gmail.com>
<mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>> wrote:

        Sure. If all you want is p-values, I'd recommend parametric
     bootstrapping (implemented in the pbkrtest package) ... that
will avoid
     these difficulties.  (I would also make sure that you know
*why* you
     want p-values on the random effects ... they have all of the
issues of
     regular p-values plus some extras:

http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects
<http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>

 <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>>

     )

     On 4/2/21 7:37 PM, Jack Solomon wrote:
      > Thanks. Just to make sure, to declare a statistically
     NON-significant
      > random effect variance component, the lower bound of the
CI must be
      > EXACTLY "0", right?
      >
      > Tha is, for example, a CI like: [.0002, .14] is a
      > statistically significant random-effect variance component but
     one that
      > perhaps borders a p-value of relatively close to but
smaller than
     .05,
      > right?
      >
      > On Fri, Apr 2, 2021 at 6:19 PM Ben Bolker
<bbolker using gmail.com <mailto:bbolker using gmail.com>
     <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>
      > <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>
<mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>>> wrote:
      >
      >         This seems like a potential can of worms (as
indeed are all
      >     hypothesis tests of null values on a boundary ...)
However,
     in this
      >     case
      >     bootstrapping (provided you have resampled appropriately -
     you may need
      >     to do hierarchical bootstrapping ...) seems reasonable,
     because a null
      >     model would give you singular fits (i.e. estimated
sd=0) half
     of the
      >     time ...
      >
      >         Happy to hear more informed opinions.
      >
      >     On 4/2/21 6:55 PM, Jack Solomon wrote:
      >      > Dear All,
      >      >
      >      > A colleague of mine suggested that I use the
bootstrapped CIs
      >     around my
      >      > model's random effect variances in place of
p-values for them.
      >      >
      >      > But random effect variances (or sds) start from
"0". So,
     to declare a
      >      > statistically NON-significant random effect variance
     component, the
      >      > lower bound of the CI must be EXACTLY "0", right?
      >      >
      >      > Thank you very much,
      >      > Jack
      >      >
      >      >       [[alternative HTML version deleted]]
      >      >
      >      > _______________________________________________
      >      > R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>
     <mailto:R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>>
      >     <mailto:R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>
     <mailto:R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>>> mailing list
      >      >
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
      >
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>>
      >      >
      >
      >     _______________________________________________
      > R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>
     <mailto:R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>>
      >     <mailto:R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>
     <mailto:R-sig-mixed-models using r-project.org
<mailto:R-sig-mixed-models using r-project.org>>> mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
      >
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>>
      >

_______________________________________________
R-sig-mixed-models using r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

	[[alternative HTML version deleted]]