[R-meta] Do we assume multi-stage sampling of effect sizes in multi-level models?

[James Pustejovsky]

Hi Fred,

This is an interesting question, for sure, and I would love to hear how
others think about it.

My own perspective: I agree with your interpretation in that the
assumptions of the multi-level meta-analysis (MLMA) model posit a two-stage
sampling process, where we first sample studies from some population of
possible studies and then sample effect sizes from the population of effect
sizes that *could have been measured* within each of those studies. The
overall average effect size parameter in the MLMA is then the average of
study-specific average effect size parameters, which in turn are averages
over a (hypothetical) set of effects that could have been assessed.

An implication of this assumption is that the MLMA model attributes
additional uncertainty to studies that measure only a single outcome. This
happens because it treats those studies as having measured just one of many
possible outcomes, rather than (for instance) as having measured the single
gold-standard outcome given the constructs/question under investigation. I
do worry about whether this assumption is reasonable, but at the moment I
don't have any great ideas about how to probe it.

Of course, just as with multi-level modeling of primary data, the
assumptions of the model don't---and needn't---necessarily match up with
the actual physical process used to collect the data. (I think this is what
you were getting at in differentiating between the epistomology and the
ontology?) Multi-level models are very commonly used with data collected
through means other than multi-stage random sampling, and I've never heard
of a meta-analytic dataset being assembled through a multi-stage sampling
of effect size information. Whether using MLMA is a reasonable statistical
strategy depends on a) whether the model's assumptions are a reasonable,
stylized approximation of the process you're investigating and b) the
robustness of the approach to violations of its assumptions.

James


On Tue, Jul 20, 2021 at 11:23 AM Farzad Keyhan <f.keyhaniha using gmail.com>
wrote:

> Hello All,
>
> Applying multi-level models to "raw data'' assumes that the data have been
> collected via a multi-stage sampling plan (e.g.,first randomly selecting
> schools, then randomly selecting students from within those selected
> schools) which makes the student data from within each school not be iid
> distributed (hierarchical dependence).
>
> But in meta-analysis, do we need to assume that a multi-stage sampling of
> "effect sizes" (first randomly selecting some studies, then selecting some
> effect sizes from within those studies) has occurred to justify the use of
> multilevel meta-regression models?
>
> I would say, epistomologically yes (but ontologically no), but I wonder
> what meta-analysis experts think?
>
> Thank you,
> Fred
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]