[R-sig-ME] Confidence interval around random effect variances in place of p-value

[Jack Solomon]

Well, how about concluding so:

If a (say 2-level) model gives a singular fit (even though perhaps there is
a "tol" that is small but not exactly "0" for that warning to show up),
that would mean we have a "practically" non-significant random-effect
variance component.



On Fri, Apr 2, 2021 at 7:15 PM Ben Bolker <bbolker using gmail.com> wrote:

>     I'm not sure that the bootstrapped CIs *wouldn't* work; they might
> return the correct proportion of singular fits ...
>
> On 4/2/21 8:12 PM, Jack Solomon wrote:
> > Thank you all very much. So, I can conclude that a likelihood ratio test
> > and/or a parametric bootstrapping can be used for random effect variance
> > component hypothesis testing.
> >
> > But I also concluded that the idea of simply using a bootstrapped CI for
> > a random-effect variance component [e.g., in lme4;
> > confint(model,method="boot",oldNames=FALSE)  ] by definition can't be
> > used for significance testing, because it requires the possibility of
> > seeing sd = 0 which can't be "strictly" captured by such a CI from a
> > multilevel model (at least not easily so).
> >
> > I hope my conclusions are correct,
> > Thank you all, Jack
> >
> > On Fri, Apr 2, 2021 at 6:51 PM Ben Bolker <bbolker using gmail.com
> > <mailto:bbolker using gmail.com>> wrote:
> >
> >        Sure. If all you want is p-values, I'd recommend parametric
> >     bootstrapping (implemented in the pbkrtest package) ... that will
> avoid
> >     these difficulties.  (I would also make sure that you know *why* you
> >     want p-values on the random effects ... they have all of the issues
> of
> >     regular p-values plus some extras:
> >
> http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects
> >     <
> http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects
> >
> >
> >     )
> >
> >     On 4/2/21 7:37 PM, Jack Solomon wrote:
> >      > Thanks. Just to make sure, to declare a statistically
> >     NON-significant
> >      > random effect variance component, the lower bound of the CI must
> be
> >      > EXACTLY "0", right?
> >      >
> >      > Tha is, for example, a CI like: [.0002, .14] is a
> >      > statistically significant random-effect variance component but
> >     one that
> >      > perhaps borders a p-value of relatively close to but smaller than
> >     .05,
> >      > right?
> >      >
> >      > On Fri, Apr 2, 2021 at 6:19 PM Ben Bolker <bbolker using gmail.com
> >     <mailto:bbolker using gmail.com>
> >      > <mailto:bbolker using gmail.com <mailto:bbolker using gmail.com>>> wrote:
> >      >
> >      >         This seems like a potential can of worms (as indeed are
> all
> >      >     hypothesis tests of null values on a boundary ...) However,
> >     in this
> >      >     case
> >      >     bootstrapping (provided you have resampled appropriately -
> >     you may need
> >      >     to do hierarchical bootstrapping ...) seems reasonable,
> >     because a null
> >      >     model would give you singular fits (i.e. estimated sd=0) half
> >     of the
> >      >     time ...
> >      >
> >      >         Happy to hear more informed opinions.
> >      >
> >      >     On 4/2/21 6:55 PM, Jack Solomon wrote:
> >      >      > Dear All,
> >      >      >
> >      >      > A colleague of mine suggested that I use the bootstrapped
> CIs
> >      >     around my
> >      >      > model's random effect variances in place of p-values for
> them.
> >      >      >
> >      >      > But random effect variances (or sds) start from "0". So,
> >     to declare a
> >      >      > statistically NON-significant random effect variance
> >     component, the
> >      >      > lower bound of the CI must be EXACTLY "0", right?
> >      >      >
> >      >      > Thank you very much,
> >      >      > Jack
> >      >      >
> >      >      >       [[alternative HTML version deleted]]
> >      >      >
> >      >      > _______________________________________________
> >      >      > R-sig-mixed-models using r-project.org
> >     <mailto:R-sig-mixed-models using r-project.org>
> >      >     <mailto:R-sig-mixed-models using r-project.org
> >     <mailto:R-sig-mixed-models using r-project.org>> mailing list
> >      >      > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> >      >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
> >      >      >
> >      >
> >      >     _______________________________________________
> >      > R-sig-mixed-models using r-project.org
> >     <mailto:R-sig-mixed-models using r-project.org>
> >      >     <mailto:R-sig-mixed-models using r-project.org
> >     <mailto:R-sig-mixed-models using r-project.org>> mailing list
> >      > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> >      >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >     <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
> >      >
> >
>

	[[alternative HTML version deleted]]