[R-meta] Negative values of df in test of moderators using robust()

[James Pustejovsky]

Hi Sebastian,

I don't think it's possible to derive a clear rule-of-thumb for this
because it depends on the configuration of the covariates (e.g., the number
of studies contributing to each category, in the example I gave), not just
on the total number of studies.

Generally, Joshi et al. found that using cluster-wild bootstrap led to
better Type-I error control and *higher power* for hypothesis tests
involving multiple constraints (that is, hypothesis tests with 2 or more
numerator degrees of freedom). Higher power is good, so it seems not
unreasonable to use CWB routinely for such tests. More pragmatically
(lazily?), I would definitely recommend using it whenever the denominator
degrees of freedom of the regular robust test are small.

James

On Wed, Feb 1, 2023 at 12:52 AM Röhl, Sebastian <
sebastian.roehl using uni-tuebingen.de> wrote:

> Dear James and Wolfgang,
>
> thank you so much for your answers! This is really helpful for me.
>
> Concerning the cluster wild bootstrapping: Do you have a rule of thumb
> below what number of studies and moderators it makes sense to use cluster
> wild bootstrapping?
>
> Best,
>
> Sebastian
>
>
>
>
>
> *Von:* James Pustejovsky <jepusto using gmail.com>
> *Gesendet:* Dienstag, 31. Januar 2023 22:08
> *An:* Viechtbauer, Wolfgang (NP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl>
> *Cc:* Röhl, Sebastian <sebastian.roehl using uni-tuebingen.de>;
> r-sig-meta-analysis using r-project.org
> *Betreff:* Re: Negative values of df in test of moderators using robust()
>
>
>
> The negative degrees of freedom arise because the small-sample
> approximation implemented in clubSandwich can become overly conservative
> when testing a hypothesis with large numerator degrees of freedom and a
> limited number of studies. For instance, suppose you are testing for
> differences in average effects between categories A, B, C, D, E, F, G, H,
> and I, so the numerator degrees of freedom will be 8 (A = B, A = C, A = D,
> etc.). If one (or more) of the categories has results from only two or
> three studies, then the denominator degrees of freedom can become negative
> and the test result should not be trusted. On the other hand, the QM test
> reported in the standard output is based on large-sample asymptotic
> approximations and should probably not be trusted either.
>
>
>
> In a recent simulation study by Megha Joshi (
> https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/),
> we found that using a cluster wild bootstrap test works much better in this
> situation. If you care about this particular test of moderators, I would
> recommend using this approach. It's implemented in the R package wildmeta:
> https://meghapsimatrix.github.io/wildmeta/
>
>
>
> James
>
>
>
> On Tue, Jan 31, 2023 at 2:50 PM Viechtbauer, Wolfgang (NP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Dear Sebastian,
>
> Yes, I assume that this is the issue. Here is a reproducible example to
> illustrate this:
>
> library(metafor)
> dat <- dat.konstantopoulos2011
> res <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat, mods = ~
> 0 + factor(year))
> robust(res, cluster=district, clubSandwich=TRUE)
>
> CC-ing James, since this is really coming from clubSandwich.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Röhl, Sebastian
> >Sent: Tuesday, 31 January, 2023 11:49
> >To: r-sig-meta-analysis using r-project.org
> >Subject: [R-meta] Negative values of df in test of moderators using
> robust()
> >
> >Dear all,
> >
> >I have a question regarding robust(): I'm using robust with
> clubSandwich-option
> >for testing moderator effects.
> >In the test of moderators, a negative df2-value appears:
> >Test of Moderators (coefficients 1:8):
> >F(df1 = 8, df2 = -2.68) = 0.0000, p-val = NA
> >
> >In the standard output (without CRVE):
> >
> >Test of Moderators (coefficients 1:8):
> >
> >QM(df = 8) = 10.0880, p-val = 0.2589
> >
> >How could I interpret this phenomen? Could this happen due to small
> numbers of ES
> >for some of the moderators?
> >
> >Thank you for your help!
> >Best,
> >Sebastian
> >
> >****************************
> >Dr. Sebastian R�hl
> >Eberhard Karls Universit�t T�bingen
> >Institute for Educational Science
> >T�bingen School of Education (T�SE)
> >Wilhelmstra�e 31 / Room 302
> >D-72074 T�bingen
> >Germany
> >
> >Phone: +49 7071 29-75527
> >Fax: +49 7071 29-35309
> >Email: sebastian.roehl using uni-tuebingen.de<mailto:
> sebastian.roehl using uni-tuebingen.de>
> >Twitter: @sebastian_roehl  @ResTeacherEdu
>
>

	[[alternative HTML version deleted]]