[R-meta] robust.rma.mv in metafor

[Reza Norouzian]

Thanks, Wolfgang and James. I just had my metafor updated and came
across the clubSandwich argument in the robust() function and was
comparing the cross-package compatibility.

Wolfgang, I think your check is sensible (perhaps improved if issued
as a warning) since perhaps the users later on want to do a QM test on
all coefficients to mimic the rma.mv() output which they'll eventually
face an error regarding the non-positive definite matrices (below).

Wald_test(res, constrain_zero(res$btt), "CR2")

Maybe in such a situation, the QM test results can be model-based and
the individual coef results can be robust although likely with dfs < 4
and of course with appropriate warnings in place.

Nice addition to the package.

Reza
Reza Norouzian (he/him/his)
Professor of TESOL | Anaheim University | Homepage
Senior Researcher for Multilingual Learners | Department of Education

Reza Norouzian (he/him/his)
Assistant Professor of TESOL | Anaheim University | Homepage
Senior Researcher for Multilingual Learners | Oregon Department of Education

On Thu, Jan 12, 2023 at 9:24 AM Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Thanks for chiming in, James.
>
> Okay, I might be inclined to remove the check when 'clubSandwich = TRUE'.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: James Pustejovsky [mailto:jepusto using gmail.com]
> >Sent: Thursday, 12 January, 2023 16:20
> >To: Viechtbauer, Wolfgang (NP)
> >Cc: R meta; Reza Norouzian
> >Subject: Re: [R-meta] robust.rma.mv in metafor
> >
> >The point about the naive-tp degrees of freedom is fair, and coef_test() should
> >probably throw an error in this case.
> >
> >The tricky thing about tests with Satterthwaite degrees of freedom is that the
> >robust SEs with the Satterthwaite df actually *can* work when the number of
> >clusters is less than the number of predictors. The validity of such tests is
> >predictor-specific: the requirement is that the predictor has within-cluster
> >variation across multiple clusters.
> >
> >Here's a (again non-sensical) example, with data chosen to ensure that there is
> >within-cluster variation in condition and word_status:
> >
> >library(metafor)
> >dat <-
> >  dat.mccurdy2020 |>
> >  subset(article %in% c(12,13,17,22,33,39,62,286,291,293,308)) |>
> >  droplevels()
> >
> >res <- rma.mv(
> >  yi = yi, V = vi,
> >  mods = ~ 0 + factor(article) + condition + word_status,
> >  random = ~ 1 | article / id,
> >  data = dat
> >)
> >
> ># produces error
> >robust(res, cluster = article, clubSandwich = TRUE)
> >
> ># throws warning for naive-tp, works for Satt
> >clubSandwich::coef_test(res, cluster = dat$article, "CR2", test=c("naive-
> >tp","Satt"))
> >
> >Best,
> >James
> >
> >On Thu, Jan 12, 2023 at 2:55 AM Viechtbauer, Wolfgang (NP)
> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >Dear Reza,
> >
> >That check is related to the default way of computing the dfs (df = n-p, where n
> >is the number of clusters and p the number of fixed effects). To see why this is
> >a problem, try:
> >
> >clubSandwich::coef_test(res, "CR2", test="naive-tp")
> >
> >and you will get negative dfs here. I am cc-ing James, because clubSandwich might
> >want to catch this. Here is a reproducible (although a bit non-sensical) example:
> >
> >library(metafor)
> >dat <- dat.konstantopoulos2011
> >res <- rma.mv(yi, vi, mods = ~ factor(district), random = ~ 1 | district/school,
> >data=dat)
> >robust(res, cluster = district)
> >clubSandwich::coef_test(res, "CR2", test="naive-tp")
> >
> >(here the dfs are 0, but the same issue).
> >
> >clubSandwich actually uses the Satterthwaite approximation for computing the dfs
> >by default and here the dfs are >= 1:
> >
> >clubSandwich::coef_test(res, "CR2")
> >
> >although they are all all quite low and so it is questionable whether those tests
> >are trustworthy in this case.
> >
> >If you still want to go ahead with this, just use coef_test(res, "CR2"). I might
> >consider taking out the check if 'clubSandwich = TRUE' in the future, although it
> >is still probably a sensible check because when n < p, the Satterthwaite
> >approximation will also tend to give rather low dfs.
> >
> >Best,
> >Wolfgang
> >
> >>-----Original Message-----
> >>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
> >>Behalf Of Reza Norouzian
> >>Sent: Thursday, 12 January, 2023 6:02
> >>To: R meta
> >>Subject: [R-meta] robust.rma.mv in metafor
> >>
> >>Dear Wolfgang,
> >>
> >>I'm trying to get the robust estimates of fixed effects for my model
> >>(res) below using robust.rma.mv().
> >>
> >>But, I wonder why I encounter an error saying:
> >>
> >>"Number of clusters (8) must be larger than the number of fixed effects (9)"
> >>
> >>Notice that when using clubSandwich directly, no errors occur.
> >>
> >>PS. Will share the data privately for reproducibility.
> >>
> >>Thank you,
> >>Reza Norouzian (he/him/his)
> >>Professor of TESOL | Anaheim University | Homepage
> >>Senior Researcher for Multilingual Learners | Department of Education
> >>###################################
> >>res <- rma.mv(lnr ~ outcome + 0, vi,
> >>       random = ~ 1|scale/study/outcome/effect,
> >>       data = dat)
> >>
> >>rob_res <- robust(res, cluster = scale, clubSandwich = TRUE)
> >>
> >>coef_test(res, "CR2")
> >>###################################