[R-meta] (Too) Many effect sizes for one single group

[Cátia Ferreira De Oliveira]

Dear James,

Just to clarify, would this still be the case even if this one paper
contained 4 experiments all on different participants? This paper would
still be the one mostly driving the nested and lack of independence in the
effect sizes.
Thank you! I may limit meta-regressions on only a very few number of
predictors then.

Best wishes,

Catia

On Mon, 10 Jan 2022 at 15:40, James Pustejovsky <jepusto using gmail.com> wrote:

> Hi Catia,
>
> Too add to the discussion, the data structure that you've described is
> one where I think some additional caution is warranted. If half of the
> effect size estimates in the synthesis come from a single paper (call
> it Paper A), whereas most of the other studies contribute just one or
> two effect sizes, then it makes me wonder whether there may be
> something qualitatively distinctive about Paper A. Are all of the
> effect sizes in Paper A equally comparable to the effect sizes from
> the other papers? Or are the effect sizes in Paper A instead covering
> sources of heterogeneity that are unexplored in the other papers?
>
> An example might make my concern a little bit clearer. Say that you're
> studying a phenomenon where there are several different scales for
> operationalizing the outcome. Say that you've got 21 studies. In 20 of
> them, the effect sizes are based one of two different outcome scales.
> But in the 21st study (Paper A), the investigators measured both of
> the commonly used outcome scales as well as 10 other scales that all
> purport to measure the construct. So with this Paper A, the effect
> sizes are heterogeneous in a way that we don't see in any of the other
> studies.
>
> I think there's a reasonable argument here that the meta-analysis
> should be conducted by first discarding the uncommon outcome scales
> from Paper A, and including only the two common scales. The scope of
> generalization is then more limited because you're looking only at
> those two scales. But generalization to other outcome scales seems
> very tenuous because there's really only one study that provides
> evidence about heterogeneity across outcomes.
>
> James
>
> On Wed, Jan 5, 2022 at 1:33 PM Cátia Ferreira De Oliveira
> <cmfo500 using york.ac.uk> wrote:
> >
> > Dear Wolfgang,
> >
> > I hope you had a lovely start to the year.
> > I am sorry for starting the year with questions, but I just wanted to
> check
> > whether there is any drawback from including a lot of effect sizes from a
> > single paper when most labs contributed to the meta-analysis with just
> one
> > or two effect sizes? This resulted in a dataset where half of the effect
> > sizes come from multiple experiments run by the same group. The nested
> > nature of the data and dependency of some effect sizes coming from the
> same
> > participants is acknowledged in the model.
> >
> > Thank you!
> >
> > Catia
> >
> > --
> > Cátia Margarida Ferreira de Oliveira
> > Psychology PhD Student
> > Department of Psychology, Room A105
> > University of York, YO10 5DD
> > Twitter: @CatiaMOliveira
> > pronouns: she, her
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>


-- 
Cátia Margarida Ferreira de Oliveira
Psychology PhD Student
Department of Psychology, Room A105
University of York, YO10 5DD
Twitter: @CatiaMOliveira
pronouns: she, her

	[[alternative HTML version deleted]]