[R-meta] robust.rma.mv in metafor

[Viechtbauer, Wolfgang (NP)]

I have now removed the check when clubSandwich=TRUE in the 'devel' version.

Best,
Wolfgang

>-----Original Message-----
>From: Reza Norouzian [mailto:rnorouzian using gmail.com]
>Sent: Thursday, 12 January, 2023 16:36
>To: Viechtbauer, Wolfgang (NP)
>Cc: James Pustejovsky; R meta
>Subject: Re: [R-meta] robust.rma.mv in metafor
>
>Thanks, Wolfgang and James. I just had my metafor updated and came
>across the clubSandwich argument in the robust() function and was
>comparing the cross-package compatibility.
>
>Wolfgang, I think your check is sensible (perhaps improved if issued
>as a warning) since perhaps the users later on want to do a QM test on
>all coefficients to mimic the rma.mv() output which they'll eventually
>face an error regarding the non-positive definite matrices (below).
>
>Wald_test(res, constrain_zero(res$btt), "CR2")
>
>Maybe in such a situation, the QM test results can be model-based and
>the individual coef results can be robust although likely with dfs < 4
>and of course with appropriate warnings in place.
>
>Nice addition to the package.
>
>Reza
>Reza Norouzian (he/him/his)
>Professor of TESOL | Anaheim University | Homepage
>Senior Researcher for Multilingual Learners | Department of Education
>
>Reza Norouzian (he/him/his)
>Assistant Professor of TESOL | Anaheim University | Homepage
>Senior Researcher for Multilingual Learners | Oregon Department of Education
>
>On Thu, Jan 12, 2023 at 9:24 AM Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>
>> Thanks for chiming in, James.
>>
>> Okay, I might be inclined to remove the check when 'clubSandwich = TRUE'.
>>
>> Best,
>> Wolfgang
>>
>> >-----Original Message-----
>> >From: James Pustejovsky [mailto:jepusto using gmail.com]
>> >Sent: Thursday, 12 January, 2023 16:20
>> >To: Viechtbauer, Wolfgang (NP)
>> >Cc: R meta; Reza Norouzian
>> >Subject: Re: [R-meta] robust.rma.mv in metafor
>> >
>> >The point about the naive-tp degrees of freedom is fair, and coef_test()
>should
>> >probably throw an error in this case.
>> >
>> >The tricky thing about tests with Satterthwaite degrees of freedom is that the
>> >robust SEs with the Satterthwaite df actually *can* work when the number of
>> >clusters is less than the number of predictors. The validity of such tests is
>> >predictor-specific: the requirement is that the predictor has within-cluster
>> >variation across multiple clusters.
>> >
>> >Here's a (again non-sensical) example, with data chosen to ensure that there
>is
>> >within-cluster variation in condition and word_status:
>> >
>> >library(metafor)
>> >dat <-
>> >  dat.mccurdy2020 |>
>> >  subset(article %in% c(12,13,17,22,33,39,62,286,291,293,308)) |>
>> >  droplevels()
>> >
>> >res <- rma.mv(
>> >  yi = yi, V = vi,
>> >  mods = ~ 0 + factor(article) + condition + word_status,
>> >  random = ~ 1 | article / id,
>> >  data = dat
>> >)
>> >
>> ># produces error
>> >robust(res, cluster = article, clubSandwich = TRUE)
>> >
>> ># throws warning for naive-tp, works for Satt
>> >clubSandwich::coef_test(res, cluster = dat$article, "CR2", test=c("naive-
>> >tp","Satt"))
>> >
>> >Best,
>> >James
>> >
>> >On Thu, Jan 12, 2023 at 2:55 AM Viechtbauer, Wolfgang (NP)
>> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >Dear Reza,
>> >
>> >That check is related to the default way of computing the dfs (df = n-p, where
>n
>> >is the number of clusters and p the number of fixed effects). To see why this
>is
>> >a problem, try:
>> >
>> >clubSandwich::coef_test(res, "CR2", test="naive-tp")
>> >
>> >and you will get negative dfs here. I am cc-ing James, because clubSandwich
>might
>> >want to catch this. Here is a reproducible (although a bit non-sensical)
>example:
>> >
>> >library(metafor)
>> >dat <- dat.konstantopoulos2011
>> >res <- rma.mv(yi, vi, mods = ~ factor(district), random = ~ 1 |
>district/school,
>> >data=dat)
>> >robust(res, cluster = district)
>> >clubSandwich::coef_test(res, "CR2", test="naive-tp")
>> >
>> >(here the dfs are 0, but the same issue).
>> >
>> >clubSandwich actually uses the Satterthwaite approximation for computing the
>dfs
>> >by default and here the dfs are >= 1:
>> >
>> >clubSandwich::coef_test(res, "CR2")
>> >
>> >although they are all all quite low and so it is questionable whether those
>tests
>> >are trustworthy in this case.
>> >
>> >If you still want to go ahead with this, just use coef_test(res, "CR2"). I
>might
>> >consider taking out the check if 'clubSandwich = TRUE' in the future, although
>it
>> >is still probably a sensible check because when n < p, the Satterthwaite
>> >approximation will also tend to give rather low dfs.
>> >
>> >Best,
>> >Wolfgang
>> >
>> >>-----Original Message-----
>> >>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org]
>On
>> >>Behalf Of Reza Norouzian
>> >>Sent: Thursday, 12 January, 2023 6:02
>> >>To: R meta
>> >>Subject: [R-meta] robust.rma.mv in metafor
>> >>
>> >>Dear Wolfgang,
>> >>
>> >>I'm trying to get the robust estimates of fixed effects for my model
>> >>(res) below using robust.rma.mv().
>> >>
>> >>But, I wonder why I encounter an error saying:
>> >>
>> >>"Number of clusters (8) must be larger than the number of fixed effects (9)"
>> >>
>> >>Notice that when using clubSandwich directly, no errors occur.
>> >>
>> >>PS. Will share the data privately for reproducibility.
>> >>
>> >>Thank you,
>> >>Reza Norouzian (he/him/his)
>> >>Professor of TESOL | Anaheim University | Homepage
>> >>Senior Researcher for Multilingual Learners | Department of Education
>> >>###################################
>> >>res <- rma.mv(lnr ~ outcome + 0, vi,
>> >>       random = ~ 1|scale/study/outcome/effect,
>> >>       data = dat)
>> >>
>> >>rob_res <- robust(res, cluster = scale, clubSandwich = TRUE)
>> >>
>> >>coef_test(res, "CR2")
>> >>###################################