[R-meta] Help specifying a three-level model

[Viechtbauer, Wolfgang (SP)]

Thanks for that reference, Frank.

I think we should have a contest as to the most complex multilevel model fitted in a meta-analysis.

My contribution: In https://link.springer.com/article/10.3758/s13423-020-01762-3 we used a 5-level model, that is, the estimates with their sampling variances at level 1, then random effects for the estimates at level 2, which in turn were nested within samples at level 3, which in turn were nested within experiments at level 4, which in turn were nested within articles at level 5. To top things off, the model also included a crossed random effect for pairs of comparisons (within experiments nested within articles). For those who are curious, the appendix shows how the data structure was coded. The actual model is essentially:

rma.mv(yi, vi, mods = ~ [...], random = list(~ 1 | article/experiment/sample/id, ~ 1 | article.experiment.pairing), data=dat)

where dat$article.experiment.pairing <- interaction(dat$article, dat$experiment, dat$pairing).

Fun stuff!

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of _mainaccount using frankbosco.com
>Sent: Thursday, 25 February, 2021 17:28
>To: r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Help specifying a three-level model
>
>Thanks, Wolfgang!
>
>This is just what we needed.
>
>Others looking for existing examples of 3-, 4-, and 5-level models: You might
>consider the following review of multi-level metas:
>
>https://doi.org/10.3758/s13428-020-01373-9
>
>(In their supplemental materials, you'll find some good sources, including a
>reference to a 4-level model reported in Psychological Bulletin
>[http://dx.doi.org/10.1037/bul0000142] whose supplemental materials, in turn,
>contain R script using metafor.)
>
>Cheers,
>
>Frank
>
>-----Original Message-----
>From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: Thursday, February 18, 2021 4:00 AM
>To: meta using frankbosco.com; r-sig-meta-analysis using r-project.org
>Subject: RE: [R-meta] Help specifying a three-level model
>
>Dear Frank,
>
>From what you describe, it sounds like
>
>random = ~ 1 | Country / SampleID / EstimateID
>
>would be a starting point, where 'EstimateID' is just
>
>dat$EstimateID <- 1:nrow(dat)
>
>Argument 'struct' is irrelevant here.
>
>Don't use -1 in 'mods'. See here:
>
>https://www.metafor-project.org/doku.php/tips:models_with_or_without_intercept
>
>With anova(Fit0, Fit1) you can do a LRT comparing the two models. For comparing
>models with different fixed effects (as you are doing), you would have to use ML
>estimation (i.e., method="ML") instead of REML estimation (which is the default).
>But you can also just do a Wald-type test instead with:
>
>anova(Fit1, btt=3:5)
>
>which should give you the omnibus test of the coefficients for HOF_powerDist,
>HOF_indivCol, HOF_masFem.
>
>To get pseudo R^2 values, you can do:
>
>(Fit0$sigma2 - Fit1$sigma2) / Fit0$sigma2
>
>to estimate the proportional reduction in each variance component as the
>additional fixed effects are added to the model or:
>
>(sum(Fit0$sigma2) - sum(Fit1$sigma2)) / sum(Fit0$sigma2)
>
>to get the proportional reduction of the total amount of heterogeneity.
>
>Given that the moderators are measured at the country level, the largest
>reductions (if any) should be observable for the country-level variance component,
>although in practice things don't divide that neatly.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis
>>[mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Frank
>>Bosco
>>Sent: Wednesday, 10 February, 2021 18:54
>>To: r-sig-meta-analysis using r-project.org
>>Subject: [R-meta] Help specifying a three-level model
>>
>>Greetings to all,
>>
>>I seek help specifying a three-level model. I suspect I need just
>>syntax help, but it might be more complicated. I’ve examined some
>>previous examples, but none seem to address my specific need. Hoping
>>there’s a three-level guru out there willing to help. Any help would be
>>greatly appreciated.
>>
>>Here’s the situation: I have 94 effect sizes pertaining to a particular
>>bivariate relation. The 94 effect sizes are nested within 56
>>independent samples. In turn, the 56 independent samples are nested
>>within 9 countries. The data file is at the URL in the script below.
>>
>>For each country, I have values for three Hofstede cultural dimensions
>>(these are continuous variables): Individualism-collectivism, power
>>distance, and masculinity/femininity. For each independent sample, I
>>have a control variable: publication year.
>>
>>I would like to ascertain the incremental variance (which, I presume,
>>would be a pseudo R^2) for the addition of the three cultural factors
>>(i.e., all three cultural factors entered simultaneously).
>>
>>At any rate, as shown below, I’m somewhat lost at the moment (I'm
>>trying to address a reviewer comment). I paste below the variable names
>>and my script skeletons.
>>
>>Thanks in advance for any help provided.
>>
>>Frank
>>
>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>
>>Variable names
>>yi
>>vi
>>SampleID
>>Country
>>PubYear
>>HOF_powerDist
>>HOF_indivCol
>>HOF_masFem
>>
>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>
>>#-- Script attempt (Fit0 contains only the control variable [PubYear];
>>Fit1 contains the control variable and the three cultural factors
>>[HOF_*])
>>
>>library(metafor)
>>dat <- read.csv("http://www.frankbosco.com/test.csv")
>>
>>Fit0 <- rma.mv(yi, vi,
>>               mods = ~ PubYear + ??? -1,     #-- (not sure whether I
>>need ‘-1’ here)
>>               random = ???,  #-- I've tried several nesting approaches
>>- some cause an error
>>               data = dat,
>>               struct = ???     #-- (Not sure what I need here)
>>)
>>
>>Fit1 <- rma.mv(yi, vi,
>>               mods = ~ PubYear + HOF_powerDist + HOF_indivCol +
>>HOF_masFem -1,      #-- (not sure whether I need ‘-1’ here)
>>               random = ???,  #-- I've tried several nesting approaches
>>- some cause an error
>>               data = dat,
>>               struct = ???      #-- (Not sure what I need here)
>>)
>>
>>anova(Fit0, Fit1)
>>
>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>
>>
>>Frank Bosco, Ph.D.
>>Director, metaBUS.org
>>Associate Professor
>>School of Business
>>Department of Management
>>
>>Virginia Commonwealth University
>>Snead Hall
>>301 West Main Street, Room B4151
>>Richmond, Virginia 23284
>>804 828-3197 • Fax: 804 828-1602
>>business.vcu.edu