[R-sig-ME] Confidence interval around random effect variances in place of p-value

[Ben Bolker]

    I'm not sure that the bootstrapped CIs *wouldn't* work; they might 
return the correct proportion of singular fits ...

On 4/2/21 8:12 PM, Jack Solomon wrote:
> Thank you all very much. So, I can conclude that a likelihood ratio test 
> and/or a parametric bootstrapping can be used for random effect variance 
> component hypothesis testing.
> 
> But I also concluded that the idea of simply using a bootstrapped CI for 
> a random-effect variance component [e.g., in lme4;  
> confint(model,method="boot",oldNames=FALSE)  ] by definition can't be 
> used for significance testing, because it requires the possibility of 
> seeing sd = 0 which can't be "strictly" captured by such a CI from a 
> multilevel model (at least not easily so).
> 
> I hope my conclusions are correct,
> Thank you all, Jack
> 
> On Fri, Apr 2, 2021 at 6:51 PM Ben Bolker <bbolker using gmail.com 
> <mailto:bbolker using gmail.com>> wrote:
> 
>        Sure. If all you want is p-values, I'd recommend parametric
>     bootstrapping (implemented in the pbkrtest package) ... that will avoid
>     these difficulties.  (I would also make sure that you know *why* you
>     want p-values on the random effects ... they have all of the issues of
>     regular p-values plus some extras:
>     http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects
>     <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>
> 
>     )
> 
>     On 4/2/21 7:37 PM, Jack Solomon wrote:
>      > Thanks. Just to make sure, to declare a statistically
>     NON-significant
>      > random effect variance component, the lower bound of the CI must be
>      > EXACTLY "0", right?
>      >
>      > Tha is, for example, a CI like: [.0002, .14] is a
>      > statistically significant random-effect variance component but
>     one that
>      > perhaps borders a p-value of relatively close to but smaller than
>     .05,
>      > right?
>      >
>      > On Fri, Apr 2, 2021 at 6:19 PM Ben Bolker <bbolker using gmail.com
>     <mailto:bbolker using gmail.com>
>      > <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>> wrote:
>      >
>      >         This seems like a potential can of worms (as indeed are all
>      >     hypothesis tests of null values on a boundary ...) However,
>     in this
>      >     case
>      >     bootstrapping (provided you have resampled appropriately -
>     you may need
>      >     to do hierarchical bootstrapping ...) seems reasonable,
>     because a null
>      >     model would give you singular fits (i.e. estimated sd=0) half
>     of the
>      >     time ...
>      >
>      >         Happy to hear more informed opinions.
>      >
>      >     On 4/2/21 6:55 PM, Jack Solomon wrote:
>      >      > Dear All,
>      >      >
>      >      > A colleague of mine suggested that I use the bootstrapped CIs
>      >     around my
>      >      > model's random effect variances in place of p-values for them.
>      >      >
>      >      > But random effect variances (or sds) start from "0". So,
>     to declare a
>      >      > statistically NON-significant random effect variance
>     component, the
>      >      > lower bound of the CI must be EXACTLY "0", right?
>      >      >
>      >      > Thank you very much,
>      >      > Jack
>      >      >
>      >      >       [[alternative HTML version deleted]]
>      >      >
>      >      > _______________________________________________
>      >      > R-sig-mixed-models using r-project.org
>     <mailto:R-sig-mixed-models using r-project.org>
>      >     <mailto:R-sig-mixed-models using r-project.org
>     <mailto:R-sig-mixed-models using r-project.org>> mailing list
>      >      > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
>      >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
>      >      >
>      >
>      >     _______________________________________________
>      > R-sig-mixed-models using r-project.org
>     <mailto:R-sig-mixed-models using r-project.org>
>      >     <mailto:R-sig-mixed-models using r-project.org
>     <mailto:R-sig-mixed-models using r-project.org>> mailing list
>      > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
>      >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
>      >
>