[R-meta] robust.rma.mv in metafor

[James Pustejovsky]

The point about the naive-tp degrees of freedom is fair, and coef_test()
should probably throw an error in this case.

The tricky thing about tests with Satterthwaite degrees of freedom is that
the robust SEs with the Satterthwaite df actually *can* work when the
number of clusters is less than the number of predictors. The validity of
such tests is predictor-specific: the requirement is that the predictor has
within-cluster variation across multiple clusters.

Here's a (again non-sensical) example, with data chosen to ensure that
there is within-cluster variation in condition and word_status:

library(metafor)
dat <-
  dat.mccurdy2020 |>
  subset(article %in% c(12,13,17,22,33,39,62,286,291,293,308)) |>
  droplevels()

res <- rma.mv(
  yi = yi, V = vi,
  mods = ~ 0 + factor(article) + condition + word_status,
  random = ~ 1 | article / id,
  data = dat
)

# produces error
robust(res, cluster = article, clubSandwich = TRUE)

# throws warning for naive-tp, works for Satt
clubSandwich::coef_test(res, cluster = dat$article, "CR2",
test=c("naive-tp","Satt"))

Best,
James

On Thu, Jan 12, 2023 at 2:55 AM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Dear Reza,
>
> That check is related to the default way of computing the dfs (df = n-p,
> where n is the number of clusters and p the number of fixed effects). To
> see why this is a problem, try:
>
> clubSandwich::coef_test(res, "CR2", test="naive-tp")
>
> and you will get negative dfs here. I am cc-ing James, because
> clubSandwich might want to catch this. Here is a reproducible (although a
> bit non-sensical) example:
>
> library(metafor)
> dat <- dat.konstantopoulos2011
> res <- rma.mv(yi, vi, mods = ~ factor(district), random = ~ 1 |
> district/school, data=dat)
> robust(res, cluster = district)
> clubSandwich::coef_test(res, "CR2", test="naive-tp")
>
> (here the dfs are 0, but the same issue).
>
> clubSandwich actually uses the Satterthwaite approximation for computing
> the dfs by default and here the dfs are >= 1:
>
> clubSandwich::coef_test(res, "CR2")
>
> although they are all all quite low and so it is questionable whether
> those tests are trustworthy in this case.
>
> If you still want to go ahead with this, just use coef_test(res, "CR2"). I
> might consider taking out the check if 'clubSandwich = TRUE' in the future,
> although it is still probably a sensible check because when n < p, the
> Satterthwaite approximation will also tend to give rather low dfs.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Reza Norouzian
> >Sent: Thursday, 12 January, 2023 6:02
> >To: R meta
> >Subject: [R-meta] robust.rma.mv in metafor
> >
> >Dear Wolfgang,
> >
> >I'm trying to get the robust estimates of fixed effects for my model
> >(res) below using robust.rma.mv().
> >
> >But, I wonder why I encounter an error saying:
> >
> >"Number of clusters (8) must be larger than the number of fixed effects
> (9)"
> >
> >Notice that when using clubSandwich directly, no errors occur.
> >
> >PS. Will share the data privately for reproducibility.
> >
> >Thank you,
> >Reza Norouzian (he/him/his)
> >Professor of TESOL | Anaheim University | Homepage
> >Senior Researcher for Multilingual Learners | Department of Education
> >###################################
> >res <- rma.mv(lnr ~ outcome + 0, vi,
> >       random = ~ 1|scale/study/outcome/effect,
> >       data = dat)
> >
> >rob_res <- robust(res, cluster = scale, clubSandwich = TRUE)
> >
> >coef_test(res, "CR2")
> >###################################
>

	[[alternative HTML version deleted]]