[R-meta] A rather general question study/effect size ratio

[Farzad Keyhan]

Oops, I forgot to add and ask this, the conceptual conflict in mind is that:

1- On the one hand, the more the studies vs. effects within studies,
the higher the generalizability of our meta-analytic results.

2- On the other hand, studies with more effects can influence the
studies with a single or fewer effects.

It seems to me 1- and 2- are at odds with each other meaning that:

A collection of studies each with a single effect gives us the most
precise meta-analytic estimates (i.e., rma() type model).

Yet, the existence of single-effect studies in a group of
multiple-effect studies reduces the precision of our meta-analytic
estimates such that multiple-effect studies should influence
single-effect studies to improve the precision of the meta-analytic
estimates (i.e., rma.mv() type model)

On Fri, Feb 4, 2022 at 12:10 PM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:
>
> Good point, Michael.
>
> However, I was thinking of this more from the perspective of sampling.
> The existence of multiple effects per study (scenarios A & B) may be
> taken to indicate that the estimates have been sampled under a
> multistage sampling plan from a larger population of studies (i.e.,
> first, studies were randomly sampled, then, from within them, some
> effects were randomly sampled, under a 3-level MLMA model).
>
> The existence of a single effect per study (scenario C) may be taken
> to indicate that the estimates have been sampled under a simple random
> sampling plan from a larger population of studies (i.e., effects were
> equally likely to be randomly sampled from any study in the population
> of studies).
>
> It would seem to me that generally the more the studies the better our
> overall sense of the overall effect in the full population of studies.
>
> Of the three scenarios (C) is the trickiest to me. It's a tough call
> to say because each study has provided a single effect, then effects
> were equally likely to be randomly sampled from any study in the
> population of studies. Maybe if that had been the case, we would have
> seen some studies with more than a single effect.
>
> Perhaps all of this takes us back to using some form of
> robust/sandwich estimation of effects, even if we have a single effect
> per study.
>
> Fred
>
> On Fri, Feb 4, 2022 at 11:19 AM Michael Dewey <lists using dewey.myzen.co.uk> wrote:
> >
> > Dear Farzad
> >
> > If having more studies means that the treatment has been tried in a
> > wider range of settings then my vote is for more studies rather than
> > more effects per study. If the settings are homogeneous but the outcomes
> > are varied then more outcomes might improve generalisability.
> >
> > Michael
> >
> > On 04/02/2022 16:50, Farzad Keyhan wrote:
> > > Hello All,
> > >
> > > All else equal, would it be more desirable to have (A) 40 studies each
> > > with two estimates of effect size or (B) 20 studies each with 4
> > > estimates of effect size?
> > >
> > > My intuition is that (A) is more desirable than (B).
> > >
> > > Would it also be more desirable to have (C) 80 studies each with 1
> > > estimate of effect size over (A) and (B)?
> > >
> > > ps. By "desirable" I mean higher generalizability to the target population.
> > >
> > > Thank you,
> > > Fred
> > >
> > > _______________________________________________
> > > R-sig-meta-analysis mailing list
> > > R-sig-meta-analysis using r-project.org
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> > >
> >
> > --
> > Michael
> > http://www.dewey.myzen.co.uk/home.html