[R-meta] (Too) Many effect sizes for one single group

[James Pustejovsky]

Hi Catia,

Too add to the discussion, the data structure that you've described is
one where I think some additional caution is warranted. If half of the
effect size estimates in the synthesis come from a single paper (call
it Paper A), whereas most of the other studies contribute just one or
two effect sizes, then it makes me wonder whether there may be
something qualitatively distinctive about Paper A. Are all of the
effect sizes in Paper A equally comparable to the effect sizes from
the other papers? Or are the effect sizes in Paper A instead covering
sources of heterogeneity that are unexplored in the other papers?

An example might make my concern a little bit clearer. Say that you're
studying a phenomenon where there are several different scales for
operationalizing the outcome. Say that you've got 21 studies. In 20 of
them, the effect sizes are based one of two different outcome scales.
But in the 21st study (Paper A), the investigators measured both of
the commonly used outcome scales as well as 10 other scales that all
purport to measure the construct. So with this Paper A, the effect
sizes are heterogeneous in a way that we don't see in any of the other
studies.

I think there's a reasonable argument here that the meta-analysis
should be conducted by first discarding the uncommon outcome scales
from Paper A, and including only the two common scales. The scope of
generalization is then more limited because you're looking only at
those two scales. But generalization to other outcome scales seems
very tenuous because there's really only one study that provides
evidence about heterogeneity across outcomes.

James

On Wed, Jan 5, 2022 at 1:33 PM Cátia Ferreira De Oliveira
<cmfo500 using york.ac.uk> wrote:
>
> Dear Wolfgang,
>
> I hope you had a lovely start to the year.
> I am sorry for starting the year with questions, but I just wanted to check
> whether there is any drawback from including a lot of effect sizes from a
> single paper when most labs contributed to the meta-analysis with just one
> or two effect sizes? This resulted in a dataset where half of the effect
> sizes come from multiple experiments run by the same group. The nested
> nature of the data and dependency of some effect sizes coming from the same
> participants is acknowledged in the model.
>
> Thank you!
>
> Catia
>
> --
> Cátia Margarida Ferreira de Oliveira
> Psychology PhD Student
> Department of Psychology, Room A105
> University of York, YO10 5DD
> Twitter: @CatiaMOliveira
> pronouns: she, her
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis