[R-meta] Do we assume multi-stage sampling of effect sizes in multi-level models?

[Farzad Keyhan]

Thanks, James. You are right about what I meant by epistemological vs.
ontological. But the problem is that in the case of "primary data", it is
"theoretically possible" to follow a multi-stage plan but in many cases we
may not "afford" to do it, and so it doesn't happen (always
epistemologically plausible, but at times ontologically implausible).

But in the case of multilevel meta-regression, it's "not even theoretically
possible" to assume so. Of course, I fully understand that there is no
remedy and it is what it is. But I just wanted to make sure I'm not way off
on this as a non-stats person.

Thank you again, for your expertise and dedication,
Fred

ps. My colleagues and I have run into a question reading your
expanding range paper (and applying it to our ongoing meta project) but
I'll, if you don't mind, ask that on this forum later.

On Wed, Jul 21, 2021 at 10:07 AM James Pustejovsky <jepusto using gmail.com>
wrote:

> Hi Fred,
>
> This is an interesting question, for sure, and I would love to hear how
> others think about it.
>
> My own perspective: I agree with your interpretation in that the
> assumptions of the multi-level meta-analysis (MLMA) model posit a two-stage
> sampling process, where we first sample studies from some population of
> possible studies and then sample effect sizes from the population of effect
> sizes that *could have been measured* within each of those studies. The
> overall average effect size parameter in the MLMA is then the average of
> study-specific average effect size parameters, which in turn are averages
> over a (hypothetical) set of effects that could have been assessed.
>
> An implication of this assumption is that the MLMA model attributes
> additional uncertainty to studies that measure only a single outcome. This
> happens because it treats those studies as having measured just one of many
> possible outcomes, rather than (for instance) as having measured the single
> gold-standard outcome given the constructs/question under investigation. I
> do worry about whether this assumption is reasonable, but at the moment I
> don't have any great ideas about how to probe it.
>
> Of course, just as with multi-level modeling of primary data, the
> assumptions of the model don't---and needn't---necessarily match up with
> the actual physical process used to collect the data. (I think this is what
> you were getting at in differentiating between the epistomology and the
> ontology?) Multi-level models are very commonly used with data collected
> through means other than multi-stage random sampling, and I've never heard
> of a meta-analytic dataset being assembled through a multi-stage sampling
> of effect size information. Whether using MLMA is a reasonable statistical
> strategy depends on a) whether the model's assumptions are a reasonable,
> stylized approximation of the process you're investigating and b) the
> robustness of the approach to violations of its assumptions.
>
> James
>
>
> On Tue, Jul 20, 2021 at 11:23 AM Farzad Keyhan <f.keyhaniha using gmail.com>
> wrote:
>
>> Hello All,
>>
>> Applying multi-level models to "raw data'' assumes that the data have been
>> collected via a multi-stage sampling plan (e.g.,first randomly selecting
>> schools, then randomly selecting students from within those selected
>> schools) which makes the student data from within each school not be iid
>> distributed (hierarchical dependence).
>>
>> But in meta-analysis, do we need to assume that a multi-stage sampling of
>> "effect sizes" (first randomly selecting some studies, then selecting some
>> effect sizes from within those studies) has occurred to justify the use of
>> multilevel meta-regression models?
>>
>> I would say, epistomologically yes (but ontologically no), but I wonder
>> what meta-analysis experts think?
>>
>> Thank you,
>> Fred
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

	[[alternative HTML version deleted]]