[R-meta] query re: multi-level analysis in metafor with missing standard errors - p.s.

[Viechtbauer, Wolfgang (SP)]

Dear Jenny,

To clarify one potential misconception: The type of meta-analytic model has no bearing on the way the standard error for an effect size measure should be computed. So it is not relevant whether you will conduct a multilevel meta-analysis or not.

Proper methods for computing the standard error of a standardized regression coefficient are discussed in these articles:

Yuan, K.-H., & Chan, W. (2011). Biases and standard errors of standardized regression coefficients. Psychometrika, 76(4), 670-690.

Jones, J. A., & Waller, N. G. (2013). Computing confidence intervals for standardized regression coefficients. Psychological Methods, 18(4), 435-453.

Jones, J. A., & Waller, N. G. (2015). The normal-theory and asymptotic distribution-free (adf) covariance matrix of standardized regression coefficients: Theoretical extensions and finite sample behavior. Psychometrika, 80(2), 365-378.

The results from Yuan and Chan (2011) are interesting, but not really usable in practice because you would essentially need access to the raw data to use their equations. Jones and Waller (2015) show how to do the same thing, but using only correlations. You can do the same thing with some functions that were recently added to metafor.

For the following to work, you need to install the 'devel' version of metafor as described here: https://github.com/wviechtb/metafor#installation

###################

library(metafor)

# unstandardized regression coefficients
res <- lm(mpg ~ cyl + disp + hp, data=mtcars)
summary(res)

# standardized regression coefficients
res <- lm(scale(mpg) ~ 0 + scale(cyl) + scale(disp) + scale(hp), data=mtcars)
summary(res)

# NOTE: The SEs given above are only appropriate if we assume that the null hypothesis is true and therefore cannot be used for a meta-analysis (even if they were reported)

# correlation matrix
R <- with(mtcars, cor(cbind(mpg, cyl, disp, hp)))
R

# show that we can get the same results as above just based on R (and n)
matreg(1, 2:4, R=R, n=nrow(mtcars))

# the SEs are still the same as before and again should not be used for a meta-analysis

# to compute SEs of the standardized regression coefficients that are appropriate to use for a meta-analysis, we first need to compute the (asymptotic) var-cov matrix of the correlation coefficients
sav <- rcalc(R, ni=nrow(mtcars))

# then we can use this to get SEs of the standardized regression coefficients
matreg(1, 2:4, R=R, V=sav$V)

###################

I think the 'fungible' package also has a function for this and may also provide the asymptotically distribution free method as described by Jones and Waller (2015). In any case, even with this approach, you need the full correlation matrix (of all predictors and the outcome). Maybe this is available from the articles (or the authors).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org]
>On Behalf Of Jennifer Oser
>Sent: Monday, 05 October, 2020 11:05
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] query re: multi-level analysis in metafor with missing
>standard errors - p.s.
>
>Hi all, I write with a p.s. to my post from last week (Sep 29) copied below
>that my co-authors and I are still trying to resolve:
>
>In our dataset, 83 of the total 185 coded effects are in this problematic
>category for multi-level analysis in metafor, i.e., a coefficient reported
>in a study's regression table as a standardized coefficient with no
>reported standard error.
>
>My guess is that this must be a common issue for applied researchers like
>myself and my co-authors who are interested in using this approach, as our
>dataset includes a nice distribution of studies according to standard
>indicators of article quality (we coded ISI impact factor quartile).
>
>As noted in my original post, we are very keen to try to resolve this asap
>as the rest of the manuscript is in final draft form and we've received
>very favorable comments on the substantive academic contribution.
>
>Any and all tips and hints are welcome!
>Jenny
>
>Dr. Jennifer Oser
>Department of Politics & Government
>Ben-Gurion University of the Negev
>https://www.jenniferoser.com/
>
>****************************************************************
>Jennifer Oser o using er @end|ng |rom po using t@bgu@@c@||
>Tue Sep 29 18:13:17 CEST 2020
>
>Hi all,
>
>My co-authors and I are conducting a multi-level meta-analysis, and I write
>with a question about calculating the sampling variance to use in the rma.mv
>argument. We followed the approach of calculating the sampling variance (v)
>by squaring the standard error, as documented in "Doing Meta-Analysis in R"
>here:
>https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/fitting-a-three-
>level-model.html
>.
>However, most of the regressions in the studies that meet our inclusion
>criteria report on standardized coefficients without reporting standard
>errors, and therefore cannot be included in the multi-level analysis using
>this approach.
>
>Our question is therefore: Is there a way to conduct a multi-level analysis
>that calculates the relevant sampling variance based on regression outputs
>that report standardized coefficients but do not report standard errors? We
>know that it is possible to do conversions for an effect size in
>non-multi-level studies, but we have not yet seen documentation on how to
>do this sort of conversion for multi-level meta-analysis based on
>regression outputs.
>
>If the answer is a definitive “No, this is not possible (at least at this
>time)” – this would also be a useful answer, as we have conducted
>vote-counting tests in our paper, and can report on those results along
>with the multi-level findings from the smaller number of effects that do
>include the necessary information.
>
>If the answer is “Yes, this is possible but it’s complicated…” - any advice
>and/or references for addressing this issue would be greatly appreciated,
>and we are happy to provide more information on the analysis as useful. We
>are in the final stages of revising a paper that has received very positive
>feedback in various conferences (on the topic of political efficacy and
>online political participation), and would be grateful for leads that would
>help us conduct as rigorous and comprehensive an analysis as possible.
>
>Best wishes,
>Jenny
>
>Dr. Jennifer Oser
>Department of Politics & Government
>Ben-Gurion University of the Negev
>https://www.jenniferoser.com/