[R-meta] How to deal with "dependent" Effect sizes?

[James Pustejovsky]

I interpreted Angeline's original message as describing the data structure
for one of the papers included in the meta-analysis, but I assume that the
meta-analysis includes more than a single paper with three samples.
Angeline, do you know (yet) the total number of papers from which you draw
effect size estimates? And the number of distinct samples reported in those
papers?

Incidentally, some colleagues and I have been looking at the techniques
that have been used in practice to conduct meta-analyses with dependent
effect sizes (across several different journals in psychology, education,
and medicine). Along the way, we're noting a number of ways in which the
reporting of such studies could be improved. One basic thing that we'd love
to see consistently reported is the total number of studies, the total
number of (independent) samples, and the total number of effect size
estimates (preferably also the range) after all inclusion/exclusion
criteria have been applied. For instance, fill in the blank:

The final sample consisted of XX effect size estimates, drawn from XX
> distinct samples, reported in XX papers/manuscripts. Each paper reported
> results from between 1 and XX samples (median = XX) and contributed between
> 1 and XX effect size estimates (median = XX).


On Mon, Feb 26, 2018 at 10:55 AM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

> For cluster-robust inference methods, there is the robust() function in
> metafor. James' clubSandwich package (https://cran.r-project.org/
> package=clubSandwich) also works nicely together with metafor. However,
> generally speaking, these methods work *asymptotically*. clubSandwich
> includes some small-sample corrections, but I doubt that James would
> advocate their use in such a small k setting. So I don't think
> cluster-robust inference methods are an appropriate way to handle the
> dependency here.
>
> What kind of 'effect sizes' are we talking about here anyway?
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
> >project.org] On Behalf Of Angeline Tsui
> >Sent: Monday, 26 February, 2018 17:27
> >To: Mark White
> >Cc: r-sig-meta-analysis at r-project.org
> >Subject: Re: [R-meta] How to deal with "dependent" Effect sizes?
> >
> >Hello Mark,
> >
> >Thanks for sharing your manuscript with me. I will take a look.
> >
> >But, if anyone knows how to deal with dependent ES using metafor, please
> >let me know.
> >
> >Best,
> >Angeline
> >
> >On Mon, Feb 26, 2018 at 10:26 AM, Mark White <markhwhiteii at gmail.com>
> >wrote:
> >
> >> I did a meta-analysis that dealt with a lot of studies with dependent
> >> variables at the participant level. I got a great deal of help from
> >this
> >> group (and others), and I settled eventually on robust variance
> >estimation.
> >> See pages 21 to 23 here (https://github.com/markhwhiteii/prej-beh-meta/
> >> blob/master/docs/manuscript.pdf) on how I came to that decision and
> >some
> >> great references for using their robumeta package. I'm sure there is a
> >way
> >> to do this in metafor, as well.
> >>
> >> On Mon, Feb 26, 2018 at 10:08 AM, Angeline Tsui
> ><angelinetsui at gmail.com>
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I am working on a meta-analysis that may contain dependent effect
> >sizes.
> >>> For example, there are five studies in a paper. However, study 1, 2
> >and 3
> >>> tested the same group of participants whereas study 4 and 5 tested
> >>> different groups of participants. This means that the effect sizes in
> >>> study
> >>> 1, 2 and 3 are dependent of each other, whereas study 4 and 5 are
> >>> independent of each other. In this case, how should I incorporate
> >these
> >>> studies in a meta-analysis? Specifically, my concern is that if I put
> >all
> >>> five studies in a meta-regression, then I am not ensuring that each
> >effect
> >>> size is independent of each other.
> >>>
> >>> Thanks,
> >>> Angeline
> >>>
> >>> --
> >>> Best Regards,
> >>> Angeline
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]