[R-meta] robust.rma.mv in metafor

[Viechtbauer, Wolfgang (NP)]

Thanks for chiming in, James.

Okay, I might be inclined to remove the check when 'clubSandwich = TRUE'.

Best,
Wolfgang

>-----Original Message-----
>From: James Pustejovsky [mailto:jepusto using gmail.com]
>Sent: Thursday, 12 January, 2023 16:20
>To: Viechtbauer, Wolfgang (NP)
>Cc: R meta; Reza Norouzian
>Subject: Re: [R-meta] robust.rma.mv in metafor
>
>The point about the naive-tp degrees of freedom is fair, and coef_test() should
>probably throw an error in this case.
>
>The tricky thing about tests with Satterthwaite degrees of freedom is that the
>robust SEs with the Satterthwaite df actually *can* work when the number of
>clusters is less than the number of predictors. The validity of such tests is
>predictor-specific: the requirement is that the predictor has within-cluster
>variation across multiple clusters.
>
>Here's a (again non-sensical) example, with data chosen to ensure that there is
>within-cluster variation in condition and word_status:
>
>library(metafor)
>dat <-
>  dat.mccurdy2020 |>
>  subset(article %in% c(12,13,17,22,33,39,62,286,291,293,308)) |>
>  droplevels()
>
>res <- rma.mv(
>  yi = yi, V = vi,
>  mods = ~ 0 + factor(article) + condition + word_status,
>  random = ~ 1 | article / id,
>  data = dat
>)
>
># produces error
>robust(res, cluster = article, clubSandwich = TRUE)
>
># throws warning for naive-tp, works for Satt
>clubSandwich::coef_test(res, cluster = dat$article, "CR2", test=c("naive-
>tp","Satt"))
>
>Best,
>James
>
>On Thu, Jan 12, 2023 at 2:55 AM Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>Dear Reza,
>
>That check is related to the default way of computing the dfs (df = n-p, where n
>is the number of clusters and p the number of fixed effects). To see why this is
>a problem, try:
>
>clubSandwich::coef_test(res, "CR2", test="naive-tp")
>
>and you will get negative dfs here. I am cc-ing James, because clubSandwich might
>want to catch this. Here is a reproducible (although a bit non-sensical) example:
>
>library(metafor)
>dat <- dat.konstantopoulos2011
>res <- rma.mv(yi, vi, mods = ~ factor(district), random = ~ 1 | district/school,
>data=dat)
>robust(res, cluster = district)
>clubSandwich::coef_test(res, "CR2", test="naive-tp")
>
>(here the dfs are 0, but the same issue).
>
>clubSandwich actually uses the Satterthwaite approximation for computing the dfs
>by default and here the dfs are >= 1:
>
>clubSandwich::coef_test(res, "CR2")
>
>although they are all all quite low and so it is questionable whether those tests
>are trustworthy in this case.
>
>If you still want to go ahead with this, just use coef_test(res, "CR2"). I might
>consider taking out the check if 'clubSandwich = TRUE' in the future, although it
>is still probably a sensible check because when n < p, the Satterthwaite
>approximation will also tend to give rather low dfs.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Reza Norouzian
>>Sent: Thursday, 12 January, 2023 6:02
>>To: R meta
>>Subject: [R-meta] robust.rma.mv in metafor
>>
>>Dear Wolfgang,
>>
>>I'm trying to get the robust estimates of fixed effects for my model
>>(res) below using robust.rma.mv().
>>
>>But, I wonder why I encounter an error saying:
>>
>>"Number of clusters (8) must be larger than the number of fixed effects (9)"
>>
>>Notice that when using clubSandwich directly, no errors occur.
>>
>>PS. Will share the data privately for reproducibility.
>>
>>Thank you,
>>Reza Norouzian (he/him/his)
>>Professor of TESOL | Anaheim University | Homepage
>>Senior Researcher for Multilingual Learners | Department of Education
>>###################################
>>res <- rma.mv(lnr ~ outcome + 0, vi,
>>       random = ~ 1|scale/study/outcome/effect,
>>       data = dat)
>>
>>rob_res <- robust(res, cluster = scale, clubSandwich = TRUE)
>>
>>coef_test(res, "CR2")
>>###################################