[R-meta] Do we assume multi-stage sampling of effect sizes in multi-level models?

[James Pustejovsky]

I'm not sure i agree about the theoretical impossibility of MLMA.

Consider that the regular old random effects model also posits that we are
sampling studies from some population. Usually that population is
hypothetical (the set of possible studies that could conceivably be
conducted on the topic). But sometimes we may identify a very large body of
literature and then literally draw a random sample of records for purposes
of coding (because coding is expensive and we have limited resources).

One could imagine doing the same in a multi-stage setting, where every
study in the literature has measured a very large number of outcomes. We
first sample studies, then (again due to resource constraints), sample only
a few of the outcomes from each study for purposes of effect size
calculation. This is less plausible as a physical process, admittedly. But
we could imagine that the primary investigators are engaging in something
akin to this when they design their primary studies. Ideally, they would
measure many different outcomes using many different
instruments/scales/whatnot. But due to resource constraints, they can
actually only collect a few measurements. Perhaps they choose
instruments/scales more or less at random?

On Wed, Jul 21, 2021 at 10:43 AM Farzad Keyhan <f.keyhaniha using gmail.com>
wrote:

> Thanks, James. You are right about what I meant by epistemological vs.
> ontological. But the problem is that in the case of "primary data", it is
> "theoretically possible" to follow a multi-stage plan but in many cases we
> may not "afford" to do it, and so it doesn't happen (always
> epistemologically plausible, but at times ontologically implausible).
>
> But in the case of multilevel meta-regression, it's "not even
> theoretically possible" to assume so. Of course, I fully understand that
> there is no remedy and it is what it is. But I just wanted to make sure I'm
> not way off on this as a non-stats person.
>
> Thank you again, for your expertise and dedication,
> Fred
>
> ps. My colleagues and I have run into a question reading your
> expanding range paper (and applying it to our ongoing meta project) but
> I'll, if you don't mind, ask that on this forum later.
>
> On Wed, Jul 21, 2021 at 10:07 AM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Hi Fred,
>>
>> This is an interesting question, for sure, and I would love to hear how
>> others think about it.
>>
>> My own perspective: I agree with your interpretation in that the
>> assumptions of the multi-level meta-analysis (MLMA) model posit a two-stage
>> sampling process, where we first sample studies from some population of
>> possible studies and then sample effect sizes from the population of effect
>> sizes that *could have been measured* within each of those studies. The
>> overall average effect size parameter in the MLMA is then the average of
>> study-specific average effect size parameters, which in turn are averages
>> over a (hypothetical) set of effects that could have been assessed.
>>
>> An implication of this assumption is that the MLMA model attributes
>> additional uncertainty to studies that measure only a single outcome. This
>> happens because it treats those studies as having measured just one of many
>> possible outcomes, rather than (for instance) as having measured the single
>> gold-standard outcome given the constructs/question under investigation. I
>> do worry about whether this assumption is reasonable, but at the moment I
>> don't have any great ideas about how to probe it.
>>
>> Of course, just as with multi-level modeling of primary data, the
>> assumptions of the model don't---and needn't---necessarily match up with
>> the actual physical process used to collect the data. (I think this is what
>> you were getting at in differentiating between the epistomology and the
>> ontology?) Multi-level models are very commonly used with data collected
>> through means other than multi-stage random sampling, and I've never heard
>> of a meta-analytic dataset being assembled through a multi-stage sampling
>> of effect size information. Whether using MLMA is a reasonable statistical
>> strategy depends on a) whether the model's assumptions are a reasonable,
>> stylized approximation of the process you're investigating and b) the
>> robustness of the approach to violations of its assumptions.
>>
>> James
>>
>>
>> On Tue, Jul 20, 2021 at 11:23 AM Farzad Keyhan <f.keyhaniha using gmail.com>
>> wrote:
>>
>>> Hello All,
>>>
>>> Applying multi-level models to "raw data'' assumes that the data have
>>> been
>>> collected via a multi-stage sampling plan (e.g.,first randomly selecting
>>> schools, then randomly selecting students from within those selected
>>> schools) which makes the student data from within each school not be iid
>>> distributed (hierarchical dependence).
>>>
>>> But in meta-analysis, do we need to assume that a multi-stage sampling of
>>> "effect sizes" (first randomly selecting some studies, then selecting
>>> some
>>> effect sizes from within those studies) has occurred to justify the use
>>> of
>>> multilevel meta-regression models?
>>>
>>> I would say, epistomologically yes (but ontologically no), but I wonder
>>> what meta-analysis experts think?
>>>
>>> Thank you,
>>> Fred
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>

	[[alternative HTML version deleted]]