[R-meta] Negative values of df in test of moderators using robust()

[Röhl, Sebastian]

Dear James and Wolfgang,
thank you so much for your answers! This is really helpful for me.
Concerning the cluster wild bootstrapping: Do you have a rule of thumb below what number of studies and moderators it makes sense to use cluster wild bootstrapping?
Best,
Sebastian


Von: James Pustejovsky <jepusto using gmail.com>
Gesendet: Dienstag, 31. Januar 2023 22:08
An: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
Cc: Röhl, Sebastian <sebastian.roehl using uni-tuebingen.de>; r-sig-meta-analysis using r-project.org
Betreff: Re: Negative values of df in test of moderators using robust()

The negative degrees of freedom arise because the small-sample approximation implemented in clubSandwich can become overly conservative when testing a hypothesis with large numerator degrees of freedom and a limited number of studies. For instance, suppose you are testing for differences in average effects between categories A, B, C, D, E, F, G, H, and I, so the numerator degrees of freedom will be 8 (A = B, A = C, A = D, etc.). If one (or more) of the categories has results from only two or three studies, then the denominator degrees of freedom can become negative and the test result should not be trusted. On the other hand, the QM test reported in the standard output is based on large-sample asymptotic approximations and should probably not be trusted either.

In a recent simulation study by Megha Joshi (https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/), we found that using a cluster wild bootstrap test works much better in this situation. If you care about this particular test of moderators, I would recommend using this approach. It's implemented in the R package wildmeta: https://meghapsimatrix.github.io/wildmeta/

James

On Tue, Jan 31, 2023 at 2:50 PM Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>> wrote:
Dear Sebastian,

Yes, I assume that this is the issue. Here is a reproducible example to illustrate this:

library(metafor)
dat <- dat.konstantopoulos2011
res <- rma.mv<http://rma.mv>(yi, vi, random = ~ 1 | district/school, data=dat, mods = ~ 0 + factor(year))
robust(res, cluster=district, clubSandwich=TRUE)

CC-ing James, since this is really coming from clubSandwich.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org<mailto:r-sig-meta-analysis-bounces using r-project.org>] On
>Behalf Of Röhl, Sebastian
>Sent: Tuesday, 31 January, 2023 11:49
>To: r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>
>Subject: [R-meta] Negative values of df in test of moderators using robust()
>
>Dear all,
>
>I have a question regarding robust(): I'm using robust with clubSandwich-option
>for testing moderator effects.
>In the test of moderators, a negative df2-value appears:
>Test of Moderators (coefficients 1:8):
>F(df1 = 8, df2 = -2.68) = 0.0000, p-val = NA
>
>In the standard output (without CRVE):
>
>Test of Moderators (coefficients 1:8):
>
>QM(df = 8) = 10.0880, p-val = 0.2589
>
>How could I interpret this phenomen? Could this happen due to small numbers of ES
>for some of the moderators?
>
>Thank you for your help!
>Best,
>Sebastian
>
>****************************
>Dr. Sebastian R�hl
>Eberhard Karls Universit�t T�bingen
>Institute for Educational Science
>T�bingen School of Education (T�SE)
>Wilhelmstra�e 31 / Room 302
>D-72074 T�bingen
>Germany
>
>Phone: +49 7071 29-75527
>Fax: +49 7071 29-35309
>Email: sebastian.roehl using uni-tuebingen.de<mailto:sebastian.roehl using uni-tuebingen.de><mailto:sebastian.roehl using uni-tuebingen.de<mailto:sebastian.roehl using uni-tuebingen.de>>
>Twitter: @sebastian_roehl  @ResTeacherEdu

	[[alternative HTML version deleted]]