[R-meta] Multilevel meta-analysis with a categorical moderator | subgroup analysis using meta-regression

[Katharina Agethen]

Thanks for the explanations and especially for pointing out the importance of profiling, Wolfgang and Reza!

I have one more question:
In my meta-analysis I have a different moderator consisting of two categories (generality = global vs. contextual). For some studies I have missing values (NAs) for the category.

I have to delete those studies before specifying the single model, right? 

df.gen <- df[!is.na(df$generality), ]

mod.generality <-  rma.mv(yi = z, 
                          V = vz, 
                          slab = auth, 
                          data = df.gen,
                          random = list(~generality | samid, ~generality | esid), 
                          test = "t", 
                          method = "REML",
                          dfs="contain",
                          mods = ~ generality-1,
                          struct = c("DIAG", "DIAG"))
profile(mod.generality)
mod.generality.robust <- robust(mod.generality, cluster=df.gen$samid, clubSandwich = TRUE)
summary(mod.generality.robust )

Thanks,
Katharina 

-----Ursprüngliche Nachricht-----
Von: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl> 
Gesendet: Thursday, April 18, 2024 12:09 PM
An: Reza Norouzian <rnorouzian using gmail.com>; Katharina Agethen <katharina.agethen using th-owl.de>
Cc: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
Betreff: RE: [R-meta] Multilevel meta-analysis with a categorical moderator | subgroup analysis using meta-regression

I wanted to double-check that the equivalence also holds when using cluster-robust variance estimation, so I ran the following example.

###################################

library(metafor)

dat <- dat.assink2016

# standard multilevel random-effects model res <- rma.mv(yi, vi, random = ~ 1 | study/id, data=dat) res

# allow the mean effect and the between- and within-study variances to differ across 'deltype'
res <- rma.mv(yi, vi, mods = ~ 0 + deltype,
              random = list(~ deltype | study, ~ deltype | id),
              struct="DIAG", data=dat)
res

# profile the variance components
profile(res)

# standard multilevel random-effects model within each deltype subset
res1 <- rma.mv(yi, vi, random = ~ 1 | study/id, data=dat, subset=deltype=="covert")
res2 <- rma.mv(yi, vi, random = ~ 1 | study/id, data=dat, subset=deltype=="general")
res3 <- rma.mv(yi, vi, random = ~ 1 | study/id, data=dat, subset=deltype=="overt")

# compare the estimates of the fixed effects round(coef(summary(res)), 4) round(coef(summary(res1)), 4) round(coef(summary(res2)), 4) round(coef(summary(res3)), 4)

# compare the estimates of the variances round(rbind(res$tau2, res$gamma2), 4) round(sapply(list(res1, res2, res3), \(x) x$sigma2), 4)

# cluster-robust variance estimation
round(coef(summary(robust(res, cluster=study, clubSandwich=TRUE))), 4) #round(coef(summary(robust(res1, cluster=study, clubSandwich=TRUE))), 4) # cannot be run round(coef(summary(robust(res2, cluster=study, clubSandwich=TRUE))), 4) round(coef(summary(robust(res3, cluster=study, clubSandwich=TRUE))), 4)

###################################

The profiling reveals that tau^2_1 is not identifiable -- the profile is entirely flat -- because there is a single study for deltype 'covert'. Note that when fitting model 'res1', rma.mv() detects this issue and sets the study-level sigma^2 equal to 0. However, this check isn't done when fitting model 'res'. So while the fixed effect estimate for deltype 'covert' is fine, the corresponding SE doesn't make sense. This goes to show that profiling is important.

The example also shows that the rest of the estimates match with the two approaches. This is also true when using cluster-robust inference methods (except that this method cannot be used on 'res1' since there is a single cluster.

Best,
Wolfgang

> -----Original Message-----
> From: Reza Norouzian <rnorouzian using gmail.com>
> Sent: Wednesday, April 17, 2024 01:34
> To: Katharina Agethen <katharina.agethen using th-owl.de>
> Cc: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r- 
> project.org>; Viechtbauer, Wolfgang (NP) 
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: Re: [R-meta] Multilevel meta-analysis with a categorical 
> moderator | subgroup analysis using meta-regression
>
> Hi Katharina,
>
> The model that I linked to in my previous response is simply 
> equivalent to (and a shortcut for) running your "full.model" three 
> times, each time setting the "subset=" argument in your rma.mv() call 
> to one of the "pertype" categories (as Wolfgang correctly noted, make sure you profile your variance estimates).
>
> This shortcut is also nice as it allows you to test hypotheses that 
> involve one or more of the "pertype" categories from a single model.
>
> Please feel free to clarify what you want to achieve and we may be 
> able to further assist you.
>
> Reza
>
> ps. It may also be useful to think about: why, in the first place, you 
> want to conduct a subgroup analysis? I usually conduct subgroup 
> analysis of the sort you inquired about when, among other things, (a) 
> the categories of the moderator of interest are theoretically quite 
> distinct, (b) I'm not willing to allow any "push-and-pull" between the 
> categories of that moderator when some of the studies contain more 
> than one category of the moderator in them, (c) there is a strong 
> tradition that requires conducting this type of subgroup analysis in 
> the relevant field of research, and (d) the subgroup shortcut model is 
> indeed empirically (e.g., via AICc) superior to its "push-and-pull" alternative (the "mod.pertype" model in your example).
>
> On Tue, Apr 16, 2024 at 11:12 AM Viechtbauer, Wolfgang (NP) 
> <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> There should be three tau^2 values and three gamma^2 values. The 
> former are the between-sample variances and the later are the within-sample variances.
>
> I would strongly suggest to run profile() on this model to check that 
> all variance components are identifiable.
>
> https://wviechtb.github.io/metafor/reference/profile.rma.html
>
> Best,
> Wolfgang
>
> > -----Original Message-----
> > From: R-sig-meta-analysis 
> > <mailto:r-sig-meta-analysis-bounces using r-project.org>
> On Behalf
> > Of Katharina Agethen via R-sig-meta-analysis
> > Sent: Tuesday, April 16, 2024 17:04
> > To: Reza Norouzian <mailto:rnorouzian using gmail.com>; R Special Interest 
> > Group for
> Meta-
> > Analysis <mailto:r-sig-meta-analysis using r-project.org>
> > Cc: Katharina Agethen <mailto:katharina.agethen using th-owl.de>
> > Subject: Re: [R-meta] Multilevel meta-analysis with a categorical 
> > moderator | subgroup analysis using meta-regression
> >
> > Hi Reza,
> >
> > Thank you so much for your prompt reply. That is really helpful to me.
> >
> > After adjusting my code, I now obtain three tau^2 values based on my 
> > three subgroups. Does it still make sense to compute the variance 
> > components for the different levels (between/within sample) so that 
> > I may report the variance components of the "overall" predictor 
> > "pertype"? If needed, how would I
> compute
> > tau^2(3) (between samples) and tau^2(2) (within samples) for the 
> > moderation analysis?
> >
> > Best,
> > Katharina