[R-meta] Combining studies reporting effects at different level of analysis/aggregation

F S crpt@f@ @ending from gm@il@com
Sun Oct 28 19:27:21 CET 2018

Dear Wolfgang,

Thanks again for your very helpful advice! I've been digging into a few
other resources that are in line with your recommendations (e.g., Hedges,
2007, 2009) and am happy with the correction approach we discussed.
However, this approach covers specifically SMDs – which was indeed the
scenario in my meta-analysis that prompted me to consult the mailing list.
I now wonder, more out of general interest, how one would go about
correcting results from aggregated/averaged data if the effect size in
question is a correlation coefficient.

The potential for bias from aggregation seems to be quite substantial in
the domain of correlations. For instance, Brand and Bradley (2012)
demonstrated that correlations of averages are upwardly biased relative to
individual-level correlations by up to 76%. However, I didn't come across
any procedures for correcting inflated correlations of group means for
inclusion in a meta-analysis. Nickerson (1995) presented a formula for the
relationship between correlations of group averages and average
within-group correlations, but what a meta-analyst would want to compute is
not the within-group correlation, but an estimate of the individual-level
correlation across groups. To complicate matters further, in addition to
inflating the magnitude of the effect size by correlating group means
instead of individual observations, there is the additional possibility
that aggregation distorts the direction of the individual-level effect
(e.g., Ostroff & Harrison, 1999).

Do you have any recommendations for how one would correct correlations of
group means to permit inclusion in a meta-analysis of individual-level
correlations? Is this possible at all?

Many thanks,

Brand, A., & Bradley, M. T. (2012). More voodoo vorrelations: When
average-based measures inflate correlations. The Journal of General
Psychology, 2012, 139, 260-272.
Hedges, L. V. (2007). Effect sizes in cluster randomized designs. Journal
of Educational and Behavioral Statistics 32, 341-370.
Hedges, L. V. (2009). Effect sizes in nested designs. In H. Cooper, L.
V.Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd
ed., pp. 337-355). New York: RussellSage.
Nickerson, C. A. E. (1995). Does willingness to pay reflect the purchase of
moral satisfaction? A reconsideration of Kahneman and Knetsch. Journal of
Environmental Economics and Management, 28, 126-133.
Ostroff C, Harrison DA. (1999). Meta-analysis, level of analysis, and best
estimates of population correlations: Cautions for interpreting
meta-analytic results in organizational behavior. Journal of Applied
Psychology, 84, 260-270.

On Sun, Oct 14, 2018 at 2:30 PM F S <crpt.fs using gmail.com> wrote:

> Dear Wolfgang,
> Thanks for clarifying -- I will attempt this approach then, and also
> include study type as a moderator, as per your recommendation.
> All the best,
> Fabian
> On Thu, Oct 11, 2018 at 1:20 PM Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> Please always cc the mailing list when replying.
>> Yes, you could also 'guestimate' the ICC and use that (and then do a
>> sensitivity analysis). Even if you do the correction, I would still
>> recommend to include study type as a moderator in the analyses.
>> Best,
>> Wolfgang
>> -----Original Message-----
>> From: F S [mailto:crpt.fs using gmail.com]
>> Sent: Thursday, 11 October, 2018 18:47
>> To: Viechtbauer, Wolfgang (SP)
>> Subject: Re: [R-meta] Combining studies reporting effects at different
>> level of analysis/aggregation
>> Hello Wolfgang,
>> Thank you for your helpful answer. I'm afraid none of the studies in
>> question report the ICC, so I guess a precise correction for the inflated d
>> won't be possible. However, would it be sensible to instead impute a value
>> for rho and perform the adjustment for the design effect using that value?
>> Ideally, one would impute ICC values lifted from studies with a similar
>> type of aggregation and similar measures, but I suppose one could also
>> perform the correction for a range of plausible values of rho and evaluate
>> the impact on the overall results via sensitivity analysis. What do you
>> think?
>> Thank you very much,
>> Fabian
>> On Fri, Oct 5, 2018 at 1:17 PM Viechtbauer, Wolfgang (SP) <
>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> Hi Fabian,
>> I don't think you have received any responses to your question so far, so
>> let me take a stab here.
>> You did not say what kind of effect size / outcome measure you want to
>> use for your meta-analysis, but if it something like a standardized mean
>> difference ('d-values'), then what you describe is definitely an issue. The
>> means (i.e., the averaged individual responses within groups) will have a
>> lower variance than the responses from individuals, leading to higher
>> d-values in studies reporting statistics based on group-level means. That
>> makes d-values from the two types of studies pretty much non-comparable. At
>> the very least, you should include study type as a moderator in all of the
>> analyses.
>> If you know the ICC of the responses within groups, then one could
>> correct for the inflation of the d-values based on the 'variance inflation
>> factor' or 'design effect'. In essence, d-values from 'group studies' are
>> then adjusted by the multiplicative factor
>> sqrt((1+(n-1)*rho)/n),
>> where n is the (average) group size and rho is the ICC. That should make
>> the d-values from the two types of studies more directly comparable. The
>> sampling variance of a d-value from a 'group study' also needs to be
>> adjusted based on the square of the multiplicative factor (this ignores the
>> uncertainty in the estimated value of the ICC, but ignoring sources of
>> uncertainty when estimating sampling variances happens all the time).
>> Best,
>> Wolfgang
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:
>> r-sig-meta-analysis-bounces using r-project.org] On Behalf Of F S
>> Sent: Tuesday, 18 September, 2018 20:47
>> To: r-sig-meta-analysis using r-project.org
>> Subject: [R-meta] Combining studies reporting effects at different level
>> of analysis/aggregation
>> I am currently working on a meta-analysis in the social sciences. All
>> studies measured the relevant outcome at the level of participants, but a
>> few studies aggregated at a higher level of analysis (e.g., groups) before
>> statistics were computed. Can these studies be meta-analyzed together?
>> More detail: The relevant outcome is a continuous measure, assessed at the
>> level of individual participants. The majority of studies report
>> statistical effects computed at the level of participants. However, in a
>> number of studies, random assignment occurred not at the participant
>> level,
>> but at the level of groups (e.g., dyads, 3-person groups, classrooms).
>> Although each of these studies did assess the outcome at the participant
>> level, just like the other studies, statistical effects are computed at
>> the
>> group level. As such, they are different from cluster-randomized studies,
>> in which randomization occurs at the group level but results are reported
>> at the individual level. By contrast, the studies in question averaged
>> individual responses within groups before computing effects with group as
>> the unit of analysis.
>> I'm not sure I can include these studies in my meta-analysis, but could
>> not
>> find much work on this question. Ostroff and Harrison (1999) focused
>> specifically on correlations computed at different levels of analysis, and
>> they make a strong case against combining ES from such studies: "the
>> obtained meta-analytic ρ̂  may not be interpretable as an estimate of any
>> population parameter because authors have cumulated studies in which
>> samples were drawn from different levels" (p. 267).
>> Can I can include these studies reporting effects from aggregated
>> observations, and if so, are there specific procedures to do so? (I'm
>> planning to use rma.mv in metafor, with cluster-robust variance
>> estimates,
>> using clubSandwich.)
>> Many thanks!
>> Fabian

	[[alternative HTML version deleted]]

More information about the R-sig-meta-analysis mailing list