[R-meta] (Too) Many effect sizes for one single group

[James Pustejovsky]

Hi Catia,

I don't think there's a general answer here. It really depends on the
specific constructs you're studying and on the details of the studies
in your synthesis.

That said, I'll keep speculating: If the only difference between the
paper that contributes most of the effect size (what I was calling
Paper A) and the remaining papers is that Paper A includes multiple
replications, then I would probably be more comfortable including all
of the effect sizes from Paper A. But often in practice, when you've
got a multiple-study paper, the different experiments reported in the
paper are testing different variations of the intervention or
different tweaks to the protocol. If that's the case, it really
depends on whether those variations are also examined in some of the
non-Paper A effect sizes.

James

On Mon, Jan 10, 2022 at 10:41 AM Cátia Ferreira De Oliveira
<cmfo500 using york.ac.uk> wrote:
>
> Dear James,
>
> Just to clarify, would this still be the case even if this one paper contained 4 experiments all on different participants? This paper would still be the one mostly driving the nested and lack of independence in the effect sizes.
> Thank you! I may limit meta-regressions on only a very few number of predictors then.
>
> Best wishes,
>
> Catia
>
> On Mon, 10 Jan 2022 at 15:40, James Pustejovsky <jepusto using gmail.com> wrote:
>>
>> Hi Catia,
>>
>> Too add to the discussion, the data structure that you've described is
>> one where I think some additional caution is warranted. If half of the
>> effect size estimates in the synthesis come from a single paper (call
>> it Paper A), whereas most of the other studies contribute just one or
>> two effect sizes, then it makes me wonder whether there may be
>> something qualitatively distinctive about Paper A. Are all of the
>> effect sizes in Paper A equally comparable to the effect sizes from
>> the other papers? Or are the effect sizes in Paper A instead covering
>> sources of heterogeneity that are unexplored in the other papers?
>>
>> An example might make my concern a little bit clearer. Say that you're
>> studying a phenomenon where there are several different scales for
>> operationalizing the outcome. Say that you've got 21 studies. In 20 of
>> them, the effect sizes are based one of two different outcome scales.
>> But in the 21st study (Paper A), the investigators measured both of
>> the commonly used outcome scales as well as 10 other scales that all
>> purport to measure the construct. So with this Paper A, the effect
>> sizes are heterogeneous in a way that we don't see in any of the other
>> studies.
>>
>> I think there's a reasonable argument here that the meta-analysis
>> should be conducted by first discarding the uncommon outcome scales
>> from Paper A, and including only the two common scales. The scope of
>> generalization is then more limited because you're looking only at
>> those two scales. But generalization to other outcome scales seems
>> very tenuous because there's really only one study that provides
>> evidence about heterogeneity across outcomes.
>>
>> James
>>
>> On Wed, Jan 5, 2022 at 1:33 PM Cátia Ferreira De Oliveira
>> <cmfo500 using york.ac.uk> wrote:
>> >
>> > Dear Wolfgang,
>> >
>> > I hope you had a lovely start to the year.
>> > I am sorry for starting the year with questions, but I just wanted to check
>> > whether there is any drawback from including a lot of effect sizes from a
>> > single paper when most labs contributed to the meta-analysis with just one
>> > or two effect sizes? This resulted in a dataset where half of the effect
>> > sizes come from multiple experiments run by the same group. The nested
>> > nature of the data and dependency of some effect sizes coming from the same
>> > participants is acknowledged in the model.
>> >
>> > Thank you!
>> >
>> > Catia
>> >
>> > --
>> > Cátia Margarida Ferreira de Oliveira
>> > Psychology PhD Student
>> > Department of Psychology, Room A105
>> > University of York, YO10 5DD
>> > Twitter: @CatiaMOliveira
>> > pronouns: she, her
>> >
>> >         [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > R-sig-meta-analysis mailing list
>> > R-sig-meta-analysis using r-project.org
>> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>
>
> --
> Cátia Margarida Ferreira de Oliveira
> Psychology PhD Student
> Department of Psychology, Room A105
> University of York, YO10 5DD
> Twitter: @CatiaMOliveira
> pronouns: she, her