[R-meta] Combining studies reporting effects at different level of analysis/aggregation

[F S]

I am currently working on a meta-analysis in the social sciences. All
studies measured the relevant outcome at the level of participants, but a
few studies aggregated at a higher level of analysis (e.g., groups) before
statistics were computed. Can these studies be meta-analyzed together?

More detail: The relevant outcome is a continuous measure, assessed at the
level of individual participants. The majority of studies report
statistical effects computed at the level of participants. However, in a
number of studies, random assignment occurred not at the participant level,
but at the level of groups (e.g., dyads, 3-person groups, classrooms).
Although each of these studies did assess the outcome at the participant
level, just like the other studies, statistical effects are computed at the
group level. As such, they are different from cluster-randomized studies,
in which randomization occurs at the group level but results are reported
at the individual level. By contrast, the studies in question averaged
individual responses within groups before computing effects with group as
the unit of analysis.

I'm not sure I can include these studies in my meta-analysis, but could not
find much work on this question. Ostroff and Harrison (1999) focused
specifically on correlations computed at different levels of analysis, and
they make a strong case against combining ES from such studies: "the
obtained meta-analytic ρ̂  may not be interpretable as an estimate of any
population parameter because authors have cumulated studies in which
samples were drawn from different levels" (p. 267).

Can I can include these studies reporting effects from aggregated
observations, and if so, are there specific procedures to do so? (I'm
planning to use rma.mv in metafor, with cluster-robust variance estimates,
using clubSandwich.)

Many thanks!
Fabian

	[[alternative HTML version deleted]]