[R-meta] How to deal with "dependent" Effect sizes?

[Angeline Tsui]

Dear James and Wolfgang,

Thank you so much for your prompt reply. In this meta-analysis, I am
talking about "cohen's d" for my effect sizes. I have a follow up question
and I wonder if you can give me some directions:

James got my message that the data structure of my meta-analysis. Indeed, I
see at least 20 to 30 studies in total (may be more, but I am not sure yet
cause I need to contact authors for missing information to estimate the
ES). The problem is that some papers reported several samples that are
dependent with each other (i.e., they were testing the same group of
participants) whereas the other papers are reporting studies that are
totally independent (i.e., testing totally different group of
participants). Thus, my concern is how to run a meta-regression (for
example, a random-effect model to estimate the average ES) when some ES in
the dataset are dependent with each other whereas other ES are independent
with each other. Should I run two meta-regression models: one for dependent
ES only and the other for independent ES only? But I really want to combine
all studies together to get a sense of the average ES across all studies?
Also, I am planning to run moderator analysis to identify how experimental
factors can explain variability across studies. So it will be most useful
if I can run meta-regression and moderator analysis using the whole data
set.

Please share your thoughts with me.

Thanks again,
Angeline

On Mon, Feb 26, 2018 at 12:19 PM, James Pustejovsky <jepusto at gmail.com>
wrote:

> I interpreted Angeline's original message as describing the data structure
> for one of the papers included in the meta-analysis, but I assume that the
> meta-analysis includes more than a single paper with three samples.
> Angeline, do you know (yet) the total number of papers from which you draw
> effect size estimates? And the number of distinct samples reported in those
> papers?
>
> Incidentally, some colleagues and I have been looking at the techniques
> that have been used in practice to conduct meta-analyses with dependent
> effect sizes (across several different journals in psychology, education,
> and medicine). Along the way, we're noting a number of ways in which the
> reporting of such studies could be improved. One basic thing that we'd love
> to see consistently reported is the total number of studies, the total
> number of (independent) samples, and the total number of effect size
> estimates (preferably also the range) after all inclusion/exclusion
> criteria have been applied. For instance, fill in the blank:
>
> The final sample consisted of XX effect size estimates, drawn from XX
>> distinct samples, reported in XX papers/manuscripts. Each paper reported
>> results from between 1 and XX samples (median = XX) and contributed between
>> 1 and XX effect size estimates (median = XX).
>
>
> On Mon, Feb 26, 2018 at 10:55 AM, Viechtbauer Wolfgang (SP) <
> wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
>
>> For cluster-robust inference methods, there is the robust() function in
>> metafor. James' clubSandwich package (https://cran.r-project.org/pa
>> ckage=clubSandwich) also works nicely together with metafor. However,
>> generally speaking, these methods work *asymptotically*. clubSandwich
>> includes some small-sample corrections, but I doubt that James would
>> advocate their use in such a small k setting. So I don't think
>> cluster-robust inference methods are an appropriate way to handle the
>> dependency here.
>>
>> What kind of 'effect sizes' are we talking about here anyway?
>>
>> Best,
>> Wolfgang
>>
>> >-----Original Message-----
>> >From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>> >project.org] On Behalf Of Angeline Tsui
>> >Sent: Monday, 26 February, 2018 17:27
>> >To: Mark White
>> >Cc: r-sig-meta-analysis at r-project.org
>> >Subject: Re: [R-meta] How to deal with "dependent" Effect sizes?
>> >
>> >Hello Mark,
>> >
>> >Thanks for sharing your manuscript with me. I will take a look.
>> >
>> >But, if anyone knows how to deal with dependent ES using metafor, please
>> >let me know.
>> >
>> >Best,
>> >Angeline
>> >
>> >On Mon, Feb 26, 2018 at 10:26 AM, Mark White <markhwhiteii at gmail.com>
>> >wrote:
>> >
>> >> I did a meta-analysis that dealt with a lot of studies with dependent
>> >> variables at the participant level. I got a great deal of help from
>> >this
>> >> group (and others), and I settled eventually on robust variance
>> >estimation.
>> >> See pages 21 to 23 here (https://github.com/markhwhite
>> ii/prej-beh-meta/
>> >> blob/master/docs/manuscript.pdf) on how I came to that decision and
>> >some
>> >> great references for using their robumeta package. I'm sure there is a
>> >way
>> >> to do this in metafor, as well.
>> >>
>> >> On Mon, Feb 26, 2018 at 10:08 AM, Angeline Tsui
>> ><angelinetsui at gmail.com>
>> >> wrote:
>> >>
>> >>> Hello all,
>> >>>
>> >>> I am working on a meta-analysis that may contain dependent effect
>> >sizes.
>> >>> For example, there are five studies in a paper. However, study 1, 2
>> >and 3
>> >>> tested the same group of participants whereas study 4 and 5 tested
>> >>> different groups of participants. This means that the effect sizes in
>> >>> study
>> >>> 1, 2 and 3 are dependent of each other, whereas study 4 and 5 are
>> >>> independent of each other. In this case, how should I incorporate
>> >these
>> >>> studies in a meta-analysis? Specifically, my concern is that if I put
>> >all
>> >>> five studies in a meta-regression, then I am not ensuring that each
>> >effect
>> >>> size is independent of each other.
>> >>>
>> >>> Thanks,
>> >>> Angeline
>> >>>
>> >>> --
>> >>> Best Regards,
>> >>> Angeline
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>
>


-- 
Best Regards,
Angeline

	[[alternative HTML version deleted]]