[R-meta] query re: multi-level analysis in metafor with missing standard errors - p.s.

[Jennifer Oser]

Excellent - this type of code is precisely what we were hoping we could
find.
With thanks for the quick and thorough reply,
Jenny

On Mon, Oct 5, 2020 at 2:24 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Dear Jenny,
>
> To clarify one potential misconception: The type of meta-analytic model
> has no bearing on the way the standard error for an effect size measure
> should be computed. So it is not relevant whether you will conduct a
> multilevel meta-analysis or not.
>
> Proper methods for computing the standard error of a standardized
> regression coefficient are discussed in these articles:
>
> Yuan, K.-H., & Chan, W. (2011). Biases and standard errors of standardized
> regression coefficients. Psychometrika, 76(4), 670-690.
>
> Jones, J. A., & Waller, N. G. (2013). Computing confidence intervals for
> standardized regression coefficients. Psychological Methods, 18(4), 435-453.
>
> Jones, J. A., & Waller, N. G. (2015). The normal-theory and asymptotic
> distribution-free (adf) covariance matrix of standardized regression
> coefficients: Theoretical extensions and finite sample behavior.
> Psychometrika, 80(2), 365-378.
>
> The results from Yuan and Chan (2011) are interesting, but not really
> usable in practice because you would essentially need access to the raw
> data to use their equations. Jones and Waller (2015) show how to do the
> same thing, but using only correlations. You can do the same thing with
> some functions that were recently added to metafor.
>
> For the following to work, you need to install the 'devel' version of
> metafor as described here:
> https://github.com/wviechtb/metafor#installation
>
> ###################
>
> library(metafor)
>
> # unstandardized regression coefficients
> res <- lm(mpg ~ cyl + disp + hp, data=mtcars)
> summary(res)
>
> # standardized regression coefficients
> res <- lm(scale(mpg) ~ 0 + scale(cyl) + scale(disp) + scale(hp),
> data=mtcars)
> summary(res)
>
> # NOTE: The SEs given above are only appropriate if we assume that the
> null hypothesis is true and therefore cannot be used for a meta-analysis
> (even if they were reported)
>
> # correlation matrix
> R <- with(mtcars, cor(cbind(mpg, cyl, disp, hp)))
> R
>
> # show that we can get the same results as above just based on R (and n)
> matreg(1, 2:4, R=R, n=nrow(mtcars))
>
> # the SEs are still the same as before and again should not be used for a
> meta-analysis
>
> # to compute SEs of the standardized regression coefficients that are
> appropriate to use for a meta-analysis, we first need to compute the
> (asymptotic) var-cov matrix of the correlation coefficients
> sav <- rcalc(R, ni=nrow(mtcars))
>
> # then we can use this to get SEs of the standardized regression
> coefficients
> matreg(1, 2:4, R=R, V=sav$V)
>
> ###################
>
> I think the 'fungible' package also has a function for this and may also
> provide the asymptotically distribution free method as described by Jones
> and Waller (2015). In any case, even with this approach, you need the full
> correlation matrix (of all predictors and the outcome). Maybe this is
> available from the articles (or the authors).
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org]
> >On Behalf Of Jennifer Oser
> >Sent: Monday, 05 October, 2020 11:05
> >To: r-sig-meta-analysis using r-project.org
> >Subject: [R-meta] query re: multi-level analysis in metafor with missing
> >standard errors - p.s.
> >
> >Hi all, I write with a p.s. to my post from last week (Sep 29) copied
> below
> >that my co-authors and I are still trying to resolve:
> >
> >In our dataset, 83 of the total 185 coded effects are in this problematic
> >category for multi-level analysis in metafor, i.e., a coefficient reported
> >in a study's regression table as a standardized coefficient with no
> >reported standard error.
> >
> >My guess is that this must be a common issue for applied researchers like
> >myself and my co-authors who are interested in using this approach, as our
> >dataset includes a nice distribution of studies according to standard
> >indicators of article quality (we coded ISI impact factor quartile).
> >
> >As noted in my original post, we are very keen to try to resolve this asap
> >as the rest of the manuscript is in final draft form and we've received
> >very favorable comments on the substantive academic contribution.
> >
> >Any and all tips and hints are welcome!
> >Jenny
> >
> >Dr. Jennifer Oser
> >Department of Politics & Government
> >Ben-Gurion University of the Negev
> >https://www.jenniferoser.com/
> >
> >****************************************************************
> >Jennifer Oser o using er @end|ng |rom po using t@bgu@@c@||
> >Tue Sep 29 18:13:17 CEST 2020
> >
> >Hi all,
> >
> >My co-authors and I are conducting a multi-level meta-analysis, and I
> write
> >with a question about calculating the sampling variance to use in the
> rma.mv
> >argument. We followed the approach of calculating the sampling variance
> (v)
> >by squaring the standard error, as documented in "Doing Meta-Analysis in
> R"
> >here:
> >
> https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/fitting-a-three-
> >level-model.html
> >.
> >However, most of the regressions in the studies that meet our inclusion
> >criteria report on standardized coefficients without reporting standard
> >errors, and therefore cannot be included in the multi-level analysis using
> >this approach.
> >
> >Our question is therefore: Is there a way to conduct a multi-level
> analysis
> >that calculates the relevant sampling variance based on regression outputs
> >that report standardized coefficients but do not report standard errors?
> We
> >know that it is possible to do conversions for an effect size in
> >non-multi-level studies, but we have not yet seen documentation on how to
> >do this sort of conversion for multi-level meta-analysis based on
> >regression outputs.
> >
> >If the answer is a definitive “No, this is not possible (at least at this
> >time)” – this would also be a useful answer, as we have conducted
> >vote-counting tests in our paper, and can report on those results along
> >with the multi-level findings from the smaller number of effects that do
> >include the necessary information.
> >
> >If the answer is “Yes, this is possible but it’s complicated…” - any
> advice
> >and/or references for addressing this issue would be greatly appreciated,
> >and we are happy to provide more information on the analysis as useful. We
> >are in the final stages of revising a paper that has received very
> positive
> >feedback in various conferences (on the topic of political efficacy and
> >online political participation), and would be grateful for leads that
> would
> >help us conduct as rigorous and comprehensive an analysis as possible.
> >
> >Best wishes,
> >Jenny
> >
> >Dr. Jennifer Oser
> >Department of Politics & Government
> >Ben-Gurion University of the Negev
> >https://www.jenniferoser.com/
>

	[[alternative HTML version deleted]]