[R-meta] robust.rma.mv in metafor

[Viechtbauer, Wolfgang (NP)]

robust() doesn't compute model-based tests (and it would be confusing if it does so for some special cases). If the QM test cannot be computed (for whatever reason), robust() is designed to catch this and issue a warning ("Could not obtain the cluster-robust omnibus Wald test (use verbose=TRUE for more details)." If one then decides to use the model-based test, one can then use the regular output from the model.

PS: Please always include the mailing list in replies.

Best,
Wolfgang

>-----Original Message-----
>From: Reza Norouzian [mailto:rnorouzian using gmail.com]
>Sent: Thursday, 12 January, 2023 17:43
>To: Viechtbauer, Wolfgang (NP)
>Subject: Re: [R-meta] robust.rma.mv in metafor
>
>Sure, but now does the user now get a model-based QM test in the
>output of robust()? Because in your code `Wald_test(x,
>constrain_zero(x$btt), "CR2")` will eventually throw an error.
>
>Reza
>
>Reza Norouzian (he/him/his)
>Professor of TESOL | Anaheim University | Homepage
>Senior Researcher for Multilingual Learners | Department of Education
>
>On Thu, Jan 12, 2023 at 10:38 AM Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>
>> I have now removed the check when clubSandwich=TRUE in the 'devel' version.
>>
>> Best,
>> Wolfgang
>>
>> >-----Original Message-----
>> >From: Reza Norouzian [mailto:rnorouzian using gmail.com]
>> >Sent: Thursday, 12 January, 2023 16:36
>> >To: Viechtbauer, Wolfgang (NP)
>> >Cc: James Pustejovsky; R meta
>> >Subject: Re: [R-meta] robust.rma.mv in metafor
>> >
>> >Thanks, Wolfgang and James. I just had my metafor updated and came
>> >across the clubSandwich argument in the robust() function and was
>> >comparing the cross-package compatibility.
>> >
>> >Wolfgang, I think your check is sensible (perhaps improved if issued
>> >as a warning) since perhaps the users later on want to do a QM test on
>> >all coefficients to mimic the rma.mv() output which they'll eventually
>> >face an error regarding the non-positive definite matrices (below).
>> >
>> >Wald_test(res, constrain_zero(res$btt), "CR2")
>> >
>> >Maybe in such a situation, the QM test results can be model-based and
>> >the individual coef results can be robust although likely with dfs < 4
>> >and of course with appropriate warnings in place.
>> >
>> >Nice addition to the package.
>> >
>> >Reza
>> >Reza Norouzian (he/him/his)
>> >Professor of TESOL | Anaheim University | Homepage
>> >Senior Researcher for Multilingual Learners | Department of Education
>> >
>> >Reza Norouzian (he/him/his)
>> >Assistant Professor of TESOL | Anaheim University | Homepage
>> >Senior Researcher for Multilingual Learners | Oregon Department of Education
>> >
>> >On Thu, Jan 12, 2023 at 9:24 AM Viechtbauer, Wolfgang (NP)
>> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >>
>> >> Thanks for chiming in, James.
>> >>
>> >> Okay, I might be inclined to remove the check when 'clubSandwich = TRUE'.
>> >>
>> >> Best,
>> >> Wolfgang
>> >>
>> >> >-----Original Message-----
>> >> >From: James Pustejovsky [mailto:jepusto using gmail.com]
>> >> >Sent: Thursday, 12 January, 2023 16:20
>> >> >To: Viechtbauer, Wolfgang (NP)
>> >> >Cc: R meta; Reza Norouzian
>> >> >Subject: Re: [R-meta] robust.rma.mv in metafor
>> >> >
>> >> >The point about the naive-tp degrees of freedom is fair, and coef_test()
>> >should
>> >> >probably throw an error in this case.
>> >> >
>> >> >The tricky thing about tests with Satterthwaite degrees of freedom is that
>the
>> >> >robust SEs with the Satterthwaite df actually *can* work when the number of
>> >> >clusters is less than the number of predictors. The validity of such tests
>is
>> >> >predictor-specific: the requirement is that the predictor has within-
>cluster
>> >> >variation across multiple clusters.
>> >> >
>> >> >Here's a (again non-sensical) example, with data chosen to ensure that
>there
>> >is
>> >> >within-cluster variation in condition and word_status:
>> >> >
>> >> >library(metafor)
>> >> >dat <-
>> >> >  dat.mccurdy2020 |>
>> >> >  subset(article %in% c(12,13,17,22,33,39,62,286,291,293,308)) |>
>> >> >  droplevels()
>> >> >
>> >> >res <- rma.mv(
>> >> >  yi = yi, V = vi,
>> >> >  mods = ~ 0 + factor(article) + condition + word_status,
>> >> >  random = ~ 1 | article / id,
>> >> >  data = dat
>> >> >)
>> >> >
>> >> ># produces error
>> >> >robust(res, cluster = article, clubSandwich = TRUE)
>> >> >
>> >> ># throws warning for naive-tp, works for Satt
>> >> >clubSandwich::coef_test(res, cluster = dat$article, "CR2", test=c("naive-
>> >> >tp","Satt"))
>> >> >
>> >> >Best,
>> >> >James
>> >> >
>> >> >On Thu, Jan 12, 2023 at 2:55 AM Viechtbauer, Wolfgang (NP)
>> >> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >> >Dear Reza,
>> >> >
>> >> >That check is related to the default way of computing the dfs (df = n-p,
>where
>> >n
>> >> >is the number of clusters and p the number of fixed effects). To see why
>this
>> >is
>> >> >a problem, try:
>> >> >
>> >> >clubSandwich::coef_test(res, "CR2", test="naive-tp")
>> >> >
>> >> >and you will get negative dfs here. I am cc-ing James, because clubSandwich
>> >might
>> >> >want to catch this. Here is a reproducible (although a bit non-sensical)
>> >example:
>> >> >
>> >> >library(metafor)
>> >> >dat <- dat.konstantopoulos2011
>> >> >res <- rma.mv(yi, vi, mods = ~ factor(district), random = ~ 1 |
>> >district/school,
>> >> >data=dat)
>> >> >robust(res, cluster = district)
>> >> >clubSandwich::coef_test(res, "CR2", test="naive-tp")
>> >> >
>> >> >(here the dfs are 0, but the same issue).
>> >> >
>> >> >clubSandwich actually uses the Satterthwaite approximation for computing
>the
>> >dfs
>> >> >by default and here the dfs are >= 1:
>> >> >
>> >> >clubSandwich::coef_test(res, "CR2")
>> >> >
>> >> >although they are all all quite low and so it is questionable whether those
>> >tests
>> >> >are trustworthy in this case.
>> >> >
>> >> >If you still want to go ahead with this, just use coef_test(res, "CR2"). I
>> >might
>> >> >consider taking out the check if 'clubSandwich = TRUE' in the future,
>although
>> >it
>> >> >is still probably a sensible check because when n < p, the Satterthwaite
>> >> >approximation will also tend to give rather low dfs.
>> >> >
>> >> >Best,
>> >> >Wolfgang
>> >> >
>> >> >>-----Original Message-----
>> >> >>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>project.org]
>> >On
>> >> >>Behalf Of Reza Norouzian
>> >> >>Sent: Thursday, 12 January, 2023 6:02
>> >> >>To: R meta
>> >> >>Subject: [R-meta] robust.rma.mv in metafor
>> >> >>
>> >> >>Dear Wolfgang,
>> >> >>
>> >> >>I'm trying to get the robust estimates of fixed effects for my model
>> >> >>(res) below using robust.rma.mv().
>> >> >>
>> >> >>But, I wonder why I encounter an error saying:
>> >> >>
>> >> >>"Number of clusters (8) must be larger than the number of fixed effects
>(9)"
>> >> >>
>> >> >>Notice that when using clubSandwich directly, no errors occur.
>> >> >>
>> >> >>PS. Will share the data privately for reproducibility.
>> >> >>
>> >> >>Thank you,
>> >> >>Reza Norouzian (he/him/his)
>> >> >>Professor of TESOL | Anaheim University | Homepage
>> >> >>Senior Researcher for Multilingual Learners | Department of Education
>> >> >>###################################
>> >> >>res <- rma.mv(lnr ~ outcome + 0, vi,
>> >> >>       random = ~ 1|scale/study/outcome/effect,
>> >> >>       data = dat)
>> >> >>
>> >> >>rob_res <- robust(res, cluster = scale, clubSandwich = TRUE)
>> >> >>
>> >> >>coef_test(res, "CR2")
>> >> >>###################################