[R-meta] Help with explaining why I should use rma.mv

Wed Feb 9 12:18:55 CET 2022

Dear Tim,

if Wolfgang thinks it's fine than I have nothing to add ;)
It looks neat.

You are probably addressing the issue of dependency (effect sizes based 
on the same participants and the need for variance-covariance-matrix in 
rma.mv) in your manuscript too, right?

One minor point: The term "multilevel meta-regression" or "multilevel 
meta-analysis" is somewhat ambigous. Even the standard random-effects 
model can be regarded as a two-level model. In the following article 
even the Fixed effects model is pictured as a two-level model (see Table 1):
Pastor, D. A., & Lazowski, R. A. (2017). On the multilevel nature of
meta-analysis: A tutorial, comparison of software programs, and
discussion of analytic choices. Multivariate Behavioral Research, 52(6),
789–804. https://doi.org/10.1080/00273171.2017.1365684

Thus, I would recommend to specify explicitly throughout the manuscript 
how many levels are assumed by using a term like three-level 
meta-analysis rather than multilevel meta-analysis.

Best,
Lukasz
-- 
Lukasz Stasielowicz
Osnabrück University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Seminarstraße 20
49074 Osnabrück (Germany)

Am 09.02.2022 um 12:00 schrieb r-sig-meta-analysis-request using r-project.org:
> Send R-sig-meta-analysis mailing list submissions to
> 	r-sig-meta-analysis using r-project.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> or, via email, send a message with subject or body 'help' to
> 	r-sig-meta-analysis-request using r-project.org
> 
> You can reach the person managing the list at
> 	r-sig-meta-analysis-owner using r-project.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-meta-analysis digest..."
> 
> 
> Today's Topics:
> 
>     1. Re: Multiverse meta-analysis (Viechtbauer, Wolfgang (SP))
>     2. Re: Help with explaining why I should use rma.mv
>        (Viechtbauer, Wolfgang (SP))
>     3. Re:  questions on some functions in metafor and clubsandwich
>        (Viechtbauer, Wolfgang (SP))
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Wed, 9 Feb 2022 07:53:46 +0000
> From: "Viechtbauer, Wolfgang (SP)"
> 	<wolfgang.viechtbauer using maastrichtuniversity.nl>
> To: Antonello Preti <antoviral using gmail.com>
> Cc: "r-sig-meta-analysis using r-project.org"
> 	<r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Multiverse meta-analysis
> Message-ID: <071eb26cf80541fe8586ab1098c8839c using UM-MAIL3214.unimaas.nl>
> Content-Type: text/plain; charset="utf-8"
> 
> Thanks.
> 
> I took a very brief look at Voracek et al. (2019). Note that GOSH plots are already implemented in metafor:
> 
> https://wviechtb.github.io/metafor/reference/gosh.html
> 
> Beyond this, one can of course do all kinds of sensitivity analyses, for example by adjusting inclusion criteria, model type, estimation method, and so on. That's all pretty standard stuff and can be easily done with a simple loop in combination with meta or metafor. I see nothing here that requires an implementation in these packages.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: Antonello Preti [mailto:antoviral using gmail.com]
>> Sent: Saturday, 05 February, 2022 20:20
>> To: Viechtbauer, Wolfgang (SP)
>> Cc: r-sig-meta-analysis using r-project.org
>> Subject: Re: [R-meta] Multiverse meta-analysis
>>
>> ATTACHMENT(S) REMOVED: Voracek Multiverse-Analysis Approach to Meta-Analysis.pdf
>> | The Multiverse of Methods.pdf | The Multiverse of Methods
>> Harder_Supplemental_Material.pdf
>>
>> Herewith enclosed is the article I was referring to.
>> There is also another article on the topic, by Jenna Harder, with the codes.
>> However, I was unable to follow the codes because I was unable to retrieve the
>> dataset (alldata3.csv).
>>
>> All the best,
>> Antonello
>>
>> Il giorno ven 4 feb 2022 alle ore 16:18 Viechtbauer, Wolfgang (SP)
>> <wolfgang.viechtbauer using maastrichtuniversity.nl> ha scritto:
>> @Antonello: Could you send that paper to me? I don't have access to it.
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>> Behalf Of Lukasz Stasielowicz
>>> Sent: Thursday, 03 February, 2022 13:02
>>> To: r-sig-meta-analysis using r-project.org
>>> Subject: Re: [R-meta] Multiverse meta-analysis
>>>
>>> Hi,
>>>
>>> there is a dedicated multiverse package, that might be of interest to
>>> you: https://cran.r-project.org/web/packages/multiverse/index.html
>>> I haven't used it myself but there are several tutorials (vignettes).
>>>
>>> Best,
>>> --
>>> Lukasz Stasielowicz
>>> Osnabrück University
>>> Institute for Psychology
>>> Research methods, psychological assessment, and evaluation
>>> Seminarstraße 20
>>> 49074 Osnabrück (Germany)
>>>
>>>> Date: Wed, 2 Feb 2022 18:02:09 +0100
>>>> From: Antonello Preti <antoviral using gmail.com>
>>>> To: r-sig-meta-analysis using r-project.org
>>>> Subject: [R-meta] Multiverse meta-analysis
>>>>
>>>> Hi everyone. We will see any soon some implementation of the so-called
>>>> multiverse meta-analysis approach in 'meta' or 'metafor'?
>>>>
>>>> Something like what was described in Voracek et al., 2019. Zeitschrift für
>>>> Psychologie, 227(1), 64–82.
>>>> https://doi.org/10.1027/2151-2604/a000357
>>>>
>>>> The codes of Voracek are not easy to follow...
>>>>
>>>> Antonello Preti
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Wed, 9 Feb 2022 07:57:02 +0000
> From: "Viechtbauer, Wolfgang (SP)"
> 	<wolfgang.viechtbauer using maastrichtuniversity.nl>
> To: Timothy MacKenzie <fswfswt using gmail.com>, R meta
> 	<r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Help with explaining why I should use rma.mv
> Message-ID: <86b84d2a5ace438282f4e6414118d0a9 using UM-MAIL3214.unimaas.nl>
> Content-Type: text/plain; charset="us-ascii"
> 
> Dear Tim,
> 
> As far as I am concerned, this is fine.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>> Behalf Of Timothy MacKenzie
>> Sent: Sunday, 06 February, 2022 17:16
>> To: R meta
>> Subject: [R-meta] Help with explaining why I should use rma.mv
>>
>> Dear Experts,
>>
>> I'm writing a quick section in my paper explaining why I used a
>> multilevel meta-regression approach (using rma.mv).
>>
>> But to simplify what that means, I get a bit stuck.
>>
>> Below is the jist of what I've written, but is this technically accurate?
>>
>> Thanks,
>> Tim M
>> #-------
>> "When studies give multiple effect sizes, then effect sizes from the
>> same studies are often viewed to be more similar to each other than
>> those from other studies.
>>
>> This fact should make our inferences about the size of treatment
>> effect more conservative. Because, the effect sizes from each study
>> don't provide independent measures of evidence. Rather, they are
>> hierarchically dependent.
>>
>> By analogy, this is similar to the issue of nested data structures
>> commonly found in educational research. For example, students (like
>> effect sizes) from the same classrooms (like studies) tend to respond
>> more similarly to an intervention resulting in hierarchical dependence
>> emerging in such data.
>> #-------
> 
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Wed, 9 Feb 2022 08:06:08 +0000
> From: "Viechtbauer, Wolfgang (SP)"
> 	<wolfgang.viechtbauer using maastrichtuniversity.nl>
> To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>,
> 	"r-sig-meta-analysis using r-project.org"
> 	<r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta]  questions on some functions in metafor and
> 	clubsandwich
> Message-ID: <791c91452d2b4030a79858d32fc878d3 using UM-MAIL3214.unimaas.nl>
> Content-Type: text/plain; charset="us-ascii"
> 
> Dear Brendan,
> 
> Please see below.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>> Behalf Of Brendan Hutchinson
>> Sent: Wednesday, 09 February, 2022 7:54
>> To: r-sig-meta-analysis using r-project.org
>> Subject: [R-meta] questions on some functions in metafor and clubsandwich
>>
>> Hi mailing list,
>>
>> Thanks in advance for any help regarding my questions - I have two and they
>> concern the metafor and clubsandwich packages, and multilevel modelling.
>>
>> 1. My first question concerns the difference between the robust() function in
>> metafor and the coef_test() function in clubsandwich - I'm a little confused as
>> to the precise difference between these. Do they not perform the same operation?
>> Is there any situations in which one would be preferred over another?
> 
> coef_test() in itself is just a function for testing coefficients. The real difference between robust() and clubSandwich is the kind of adjustments they provide for the var-cov matrix and how they estimate the dfs. Note that metafor can now directly interface with clubSandwich. See:
> 
> See: https://wviechtb.github.io/metafor/reference/robust.html
> 
>> 2. Second, in order to control for correlated effect sizes and correlated
>> sampling variance in my own dataset, I will need to produce a variance-covariance
>> matrix for my data using the impute_covariance_matrix() function in clubsandwich,
>> which will then be fed into a multilevel model (effect sizes nested within
>> studies) specified in the metafor function rma.mv().
>>
>> My question here concerns the "smooth_vi" input of the impute_covariance_matrix()
>> function. I am a little unclear as to its use. The help page specifies "If
>> smooth_vi = TRUE, then all of the variances within cluster j will be set equal to
>> the average variance of cluster j".
>>
>> I interpreted this as though it is simply removing variance within clusters (i.e.
>> studies) via averaging, which I suspect would be inappropriate for a multi-level
>> meta-analysis in which we would want to capture that variance - indeed, is this
>> not the reason we specify a multilevel structure in the first place? What is
>> confusing to me is the only example code I have seen online appears to set
>> smooth_VI to true when specifying a multi-level model (in which effects are
>> nested within studies), so I am a little lost.
> 
> I think you are misunderstanding this option. Say you have two effect sizes with sampling variances equal to .01 and .03 within a cluster. Then with smooth_vi=TRUE, the sampling variances would be set to .02 and .02 for the two estimates.
> 
>> Once again, any help on the above is greatly appreciated!
>>
>> Brendan
> 
> 
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 
> 
> ------------------------------
> 
> End of R-sig-meta-analysis Digest, Vol 57, Issue 16
> ***************************************************