[R-meta] Negative values of df in test of moderators using robust()

[James Pustejovsky]

The negative degrees of freedom arise because the small-sample
approximation implemented in clubSandwich can become overly conservative
when testing a hypothesis with large numerator degrees of freedom and a
limited number of studies. For instance, suppose you are testing for
differences in average effects between categories A, B, C, D, E, F, G, H,
and I, so the numerator degrees of freedom will be 8 (A = B, A = C, A = D,
etc.). If one (or more) of the categories has results from only two or
three studies, then the denominator degrees of freedom can become negative
and the test result should not be trusted. On the other hand, the QM test
reported in the standard output is based on large-sample asymptotic
approximations and should probably not be trusted either.

In a recent simulation study by Megha Joshi (
https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/),
we found that using a cluster wild bootstrap test works much better in this
situation. If you care about this particular test of moderators, I would
recommend using this approach. It's implemented in the R package wildmeta:
https://meghapsimatrix.github.io/wildmeta/

James

On Tue, Jan 31, 2023 at 2:50 PM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Dear Sebastian,
>
> Yes, I assume that this is the issue. Here is a reproducible example to
> illustrate this:
>
> library(metafor)
> dat <- dat.konstantopoulos2011
> res <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat, mods = ~
> 0 + factor(year))
> robust(res, cluster=district, clubSandwich=TRUE)
>
> CC-ing James, since this is really coming from clubSandwich.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Röhl, Sebastian
> >Sent: Tuesday, 31 January, 2023 11:49
> >To: r-sig-meta-analysis using r-project.org
> >Subject: [R-meta] Negative values of df in test of moderators using
> robust()
> >
> >Dear all,
> >
> >I have a question regarding robust(): I'm using robust with
> clubSandwich-option
> >for testing moderator effects.
> >In the test of moderators, a negative df2-value appears:
> >Test of Moderators (coefficients 1:8):
> >F(df1 = 8, df2 = -2.68) = 0.0000, p-val = NA
> >
> >In the standard output (without CRVE):
> >
> >Test of Moderators (coefficients 1:8):
> >
> >QM(df = 8) = 10.0880, p-val = 0.2589
> >
> >How could I interpret this phenomen? Could this happen due to small
> numbers of ES
> >for some of the moderators?
> >
> >Thank you for your help!
> >Best,
> >Sebastian
> >
> >****************************
> >Dr. Sebastian R�hl
> >Eberhard Karls Universit�t T�bingen
> >Institute for Educational Science
> >T�bingen School of Education (T�SE)
> >Wilhelmstra�e 31 / Room 302
> >D-72074 T�bingen
> >Germany
> >
> >Phone: +49 7071 29-75527
> >Fax: +49 7071 29-35309
> >Email: sebastian.roehl using uni-tuebingen.de<mailto:
> sebastian.roehl using uni-tuebingen.de>
> >Twitter: @sebastian_roehl  @ResTeacherEdu
>

	[[alternative HTML version deleted]]