[R-meta] How to deal with "dependent" Effect sizes?

Mon Feb 26 19:33:04 CET 2018

Angeline,

My generic suggestion would be to do something like the following:

1. Either find information or make an assumption about the degree of
dependence among the effect sizes from the same sample, and then use this
to construct a "working" variance-covariance matrix for the effect size
estimates (see here for more information:
http://jepusto.github.io/imputing-covariance-matrices-for-multi-variate-meta-analysis
).
2. Use rma.mv to estimate the overall average ES and any meta-regressions
of interest. In rma.mv, you should definitely include a random effect for
each sample. You might also want to examine whether there is further
dependence among samples nested within studies, by including a random
effect for each study.
3. Once you've estimated the model with rma.mv, use the functions mentioned
above to compute robust variance estimates (RVE), clustering at the level
of studies. Using RVE will ensure that the standard errors, hypothesis
tests, and CIs for the overall average effect (and/or meta-regression
coefficient estimates) are robust to the possibility that the "working"
variance-covariance matrix is inaccurate.

James

On Mon, Feb 26, 2018 at 11:29 AM, Angeline Tsui <angelinetsui at gmail.com>
wrote:

> Dear James and Wolfgang,
>
> Thank you so much for your prompt reply. In this meta-analysis, I am
> talking about "cohen's d" for my effect sizes. I have a follow up question
> and I wonder if you can give me some directions:
>
> James got my message that the data structure of my meta-analysis. Indeed,
> I see at least 20 to 30 studies in total (may be more, but I am not sure
> yet cause I need to contact authors for missing information to estimate the
> ES). The problem is that some papers reported several samples that are
> dependent with each other (i.e., they were testing the same group of
> participants) whereas the other papers are reporting studies that are
> totally independent (i.e., testing totally different group of
> participants). Thus, my concern is how to run a meta-regression (for
> example, a random-effect model to estimate the average ES) when some ES in
> the dataset are dependent with each other whereas other ES are independent
> with each other. Should I run two meta-regression models: one for dependent
> ES only and the other for independent ES only? But I really want to combine
> all studies together to get a sense of the average ES across all studies?
> Also, I am planning to run moderator analysis to identify how experimental
> factors can explain variability across studies. So it will be most useful
> if I can run meta-regression and moderator analysis using the whole data
> set.
>
> Please share your thoughts with me.
>
> Thanks again,
> Angeline
>
> On Mon, Feb 26, 2018 at 12:19 PM, James Pustejovsky <jepusto at gmail.com>
> wrote:
>
>> I interpreted Angeline's original message as describing the data
>> structure for one of the papers included in the meta-analysis, but I assume
>> that the meta-analysis includes more than a single paper with three
>> samples. Angeline, do you know (yet) the total number of papers from which
>> you draw effect size estimates? And the number of distinct samples reported
>> in those papers?
>>
>> Incidentally, some colleagues and I have been looking at the techniques
>> that have been used in practice to conduct meta-analyses with dependent
>> effect sizes (across several different journals in psychology, education,
>> and medicine). Along the way, we're noting a number of ways in which the
>> reporting of such studies could be improved. One basic thing that we'd love
>> to see consistently reported is the total number of studies, the total
>> number of (independent) samples, and the total number of effect size
>> estimates (preferably also the range) after all inclusion/exclusion
>> criteria have been applied. For instance, fill in the blank:
>>
>> The final sample consisted of XX effect size estimates, drawn from XX
>>> distinct samples, reported in XX papers/manuscripts. Each paper reported
>>> results from between 1 and XX samples (median = XX) and contributed between
>>> 1 and XX effect size estimates (median = XX).
>>
>>
>> On Mon, Feb 26, 2018 at 10:55 AM, Viechtbauer Wolfgang (SP) <
>> wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
>>
>>> For cluster-robust inference methods, there is the robust() function in
>>> metafor. James' clubSandwich package (https://cran.r-project.org/pa
>>> ckage=clubSandwich) also works nicely together with metafor. However,
>>> generally speaking, these methods work *asymptotically*. clubSandwich
>>> includes some small-sample corrections, but I doubt that James would
>>> advocate their use in such a small k setting. So I don't think
>>> cluster-robust inference methods are an appropriate way to handle the
>>> dependency here.
>>>
>>> What kind of 'effect sizes' are we talking about here anyway?
>>>
>>> Best,
>>> Wolfgang
>>>
>>> >-----Original Message-----
>>> >From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>>> >project.org] On Behalf Of Angeline Tsui
>>> >Sent: Monday, 26 February, 2018 17:27
>>> >To: Mark White
>>> >Cc: r-sig-meta-analysis at r-project.org
>>> >Subject: Re: [R-meta] How to deal with "dependent" Effect sizes?
>>> >
>>> >Hello Mark,
>>> >
>>> >Thanks for sharing your manuscript with me. I will take a look.
>>> >
>>> >But, if anyone knows how to deal with dependent ES using metafor, please
>>> >let me know.
>>> >
>>> >Best,
>>> >Angeline
>>> >
>>> >On Mon, Feb 26, 2018 at 10:26 AM, Mark White <markhwhiteii at gmail.com>
>>> >wrote:
>>> >
>>> >> I did a meta-analysis that dealt with a lot of studies with dependent
>>> >> variables at the participant level. I got a great deal of help from
>>> >this
>>> >> group (and others), and I settled eventually on robust variance
>>> >estimation.
>>> >> See pages 21 to 23 here (https://github.com/markhwhite
>>> ii/prej-beh-meta/
>>> >> blob/master/docs/manuscript.pdf) on how I came to that decision and
>>> >some
>>> >> great references for using their robumeta package. I'm sure there is a
>>> >way
>>> >> to do this in metafor, as well.
>>> >>
>>> >> On Mon, Feb 26, 2018 at 10:08 AM, Angeline Tsui
>>> ><angelinetsui at gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Hello all,
>>> >>>
>>> >>> I am working on a meta-analysis that may contain dependent effect
>>> >sizes.
>>> >>> For example, there are five studies in a paper. However, study 1, 2
>>> >and 3
>>> >>> tested the same group of participants whereas study 4 and 5 tested
>>> >>> different groups of participants. This means that the effect sizes in
>>> >>> study
>>> >>> 1, 2 and 3 are dependent of each other, whereas study 4 and 5 are
>>> >>> independent of each other. In this case, how should I incorporate
>>> >these
>>> >>> studies in a meta-analysis? Specifically, my concern is that if I put
>>> >all
>>> >>> five studies in a meta-regression, then I am not ensuring that each
>>> >effect
>>> >>> size is independent of each other.
>>> >>>
>>> >>> Thanks,
>>> >>> Angeline
>>> >>>
>>> >>> --
>>> >>> Best Regards,
>>> >>> Angeline
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>>
>
>
> --
> Best Regards,
> Angeline
>

	[[alternative HTML version deleted]]