[R-meta] Combining studies reporting effects at different level of analysis/aggregation

[James Pustejovsky]

Fabian,

As far as I can see, it is not generally possible to recover
individual-level correlations from group averages. As Ostroff & Harrison
(among others) so, the overall individual-level correlation is a weighted
average of the within-group correlation and the correlation of group
averages. The correlation of group averages is not informative about the
within-group correlation (the two could even be of opposite signs, as in
Simpson's paradox), and so without outside information or very strong
assumptions, I see no way to figure out the within-group correlation
contribution to the overall individual-level correlation.

As an aside, even though in practice most meta-analysts probably focus most
on the overall individual-level correlation, I would argue that we should
give greater consideration to parsing out within-group correlations from
group-average correlations. Because the two parameters may be very
different---and because they may have quite distinct substantive
interpretations---I think that ideally we should be to try and conduct
syntheses that also keep them distinct (e.g., conducting separate
meta-analyses of the two types of correlations).  Of course this may be
difficult or impossible in practice due to the limitations of primary study
reporting. C'est la vie d'un meta-analyste...

James

On Sun, Oct 28, 2018 at 1:27 PM F S <crpt.fs using gmail.com> wrote:

> Dear Wolfgang,
>
> Thanks again for your very helpful advice! I've been digging into a few
> other resources that are in line with your recommendations (e.g., Hedges,
> 2007, 2009) and am happy with the correction approach we discussed.
> However, this approach covers specifically SMDs – which was indeed the
> scenario in my meta-analysis that prompted me to consult the mailing list.
> I now wonder, more out of general interest, how one would go about
> correcting results from aggregated/averaged data if the effect size in
> question is a correlation coefficient.
>
> The potential for bias from aggregation seems to be quite substantial in
> the domain of correlations. For instance, Brand and Bradley (2012)
> demonstrated that correlations of averages are upwardly biased relative to
> individual-level correlations by up to 76%. However, I didn't come across
> any procedures for correcting inflated correlations of group means for
> inclusion in a meta-analysis. Nickerson (1995) presented a formula for the
> relationship between correlations of group averages and average
> within-group correlations, but what a meta-analyst would want to compute is
> not the within-group correlation, but an estimate of the individual-level
> correlation across groups. To complicate matters further, in addition to
> inflating the magnitude of the effect size by correlating group means
> instead of individual observations, there is the additional possibility
> that aggregation distorts the direction of the individual-level effect
> (e.g., Ostroff & Harrison, 1999).
>
> Do you have any recommendations for how one would correct correlations of
> group means to permit inclusion in a meta-analysis of individual-level
> correlations? Is this possible at all?
>
> Many thanks,
> Fabian
>
> ---
> References:
> Brand, A., & Bradley, M. T. (2012). More voodoo vorrelations: When
> average-based measures inflate correlations. The Journal of General
> Psychology, 2012, 139, 260-272.
> Hedges, L. V. (2007). Effect sizes in cluster randomized designs. Journal
> of Educational and Behavioral Statistics 32, 341-370.
> Hedges, L. V. (2009). Effect sizes in nested designs. In H. Cooper, L.
> V.Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd
> ed., pp. 337-355). New York: RussellSage.
> Nickerson, C. A. E. (1995). Does willingness to pay reflect the purchase of
> moral satisfaction? A reconsideration of Kahneman and Knetsch. Journal of
> Environmental Economics and Management, 28, 126-133.
> Ostroff C, Harrison DA. (1999). Meta-analysis, level of analysis, and best
> estimates of population correlations: Cautions for interpreting
> meta-analytic results in organizational behavior. Journal of Applied
> Psychology, 84, 260-270.
>
>
>
>
>
>
> On Sun, Oct 14, 2018 at 2:30 PM F S <crpt.fs using gmail.com> wrote:
>
> > Dear Wolfgang,
> > Thanks for clarifying -- I will attempt this approach then, and also
> > include study type as a moderator, as per your recommendation.
> > All the best,
> > Fabian
> >
> > On Thu, Oct 11, 2018 at 1:20 PM Viechtbauer, Wolfgang (SP) <
> > wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >
> >> Please always cc the mailing list when replying.
> >>
> >> Yes, you could also 'guestimate' the ICC and use that (and then do a
> >> sensitivity analysis). Even if you do the correction, I would still
> >> recommend to include study type as a moderator in the analyses.
> >>
> >> Best,
> >> Wolfgang
> >>
> >> -----Original Message-----
> >> From: F S [mailto:crpt.fs using gmail.com]
> >> Sent: Thursday, 11 October, 2018 18:47
> >> To: Viechtbauer, Wolfgang (SP)
> >> Subject: Re: [R-meta] Combining studies reporting effects at different
> >> level of analysis/aggregation
> >>
> >> Hello Wolfgang,
> >>
> >> Thank you for your helpful answer. I'm afraid none of the studies in
> >> question report the ICC, so I guess a precise correction for the
> inflated d
> >> won't be possible. However, would it be sensible to instead impute a
> value
> >> for rho and perform the adjustment for the design effect using that
> value?
> >> Ideally, one would impute ICC values lifted from studies with a similar
> >> type of aggregation and similar measures, but I suppose one could also
> >> perform the correction for a range of plausible values of rho and
> evaluate
> >> the impact on the overall results via sensitivity analysis. What do you
> >> think?
> >>
> >> Thank you very much,
> >> Fabian
> >>
> >> On Fri, Oct 5, 2018 at 1:17 PM Viechtbauer, Wolfgang (SP) <
> >> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >> Hi Fabian,
> >>
> >> I don't think you have received any responses to your question so far,
> so
> >> let me take a stab here.
> >>
> >> You did not say what kind of effect size / outcome measure you want to
> >> use for your meta-analysis, but if it something like a standardized mean
> >> difference ('d-values'), then what you describe is definitely an issue.
> The
> >> means (i.e., the averaged individual responses within groups) will have
> a
> >> lower variance than the responses from individuals, leading to higher
> >> d-values in studies reporting statistics based on group-level means.
> That
> >> makes d-values from the two types of studies pretty much
> non-comparable. At
> >> the very least, you should include study type as a moderator in all of
> the
> >> analyses.
> >>
> >> If you know the ICC of the responses within groups, then one could
> >> correct for the inflation of the d-values based on the 'variance
> inflation
> >> factor' or 'design effect'. In essence, d-values from 'group studies'
> are
> >> then adjusted by the multiplicative factor
> >>
> >> sqrt((1+(n-1)*rho)/n),
> >>
> >> where n is the (average) group size and rho is the ICC. That should make
> >> the d-values from the two types of studies more directly comparable. The
> >> sampling variance of a d-value from a 'group study' also needs to be
> >> adjusted based on the square of the multiplicative factor (this ignores
> the
> >> uncertainty in the estimated value of the ICC, but ignoring sources of
> >> uncertainty when estimating sampling variances happens all the time).
> >>
> >> Best,
> >> Wolfgang
> >>
> >> -----Original Message-----
> >> From: R-sig-meta-analysis [mailto:
> >> r-sig-meta-analysis-bounces using r-project.org] On Behalf Of F S
> >> Sent: Tuesday, 18 September, 2018 20:47
> >> To: r-sig-meta-analysis using r-project.org
> >> Subject: [R-meta] Combining studies reporting effects at different level
> >> of analysis/aggregation
> >>
> >> I am currently working on a meta-analysis in the social sciences. All
> >> studies measured the relevant outcome at the level of participants, but
> a
> >> few studies aggregated at a higher level of analysis (e.g., groups)
> before
> >> statistics were computed. Can these studies be meta-analyzed together?
> >>
> >> More detail: The relevant outcome is a continuous measure, assessed at
> the
> >> level of individual participants. The majority of studies report
> >> statistical effects computed at the level of participants. However, in a
> >> number of studies, random assignment occurred not at the participant
> >> level,
> >> but at the level of groups (e.g., dyads, 3-person groups, classrooms).
> >> Although each of these studies did assess the outcome at the participant
> >> level, just like the other studies, statistical effects are computed at
> >> the
> >> group level. As such, they are different from cluster-randomized
> studies,
> >> in which randomization occurs at the group level but results are
> reported
> >> at the individual level. By contrast, the studies in question averaged
> >> individual responses within groups before computing effects with group
> as
> >> the unit of analysis.
> >>
> >> I'm not sure I can include these studies in my meta-analysis, but could
> >> not
> >> find much work on this question. Ostroff and Harrison (1999) focused
> >> specifically on correlations computed at different levels of analysis,
> and
> >> they make a strong case against combining ES from such studies: "the
> >> obtained meta-analytic ρ̂  may not be interpretable as an estimate of
> any
> >> population parameter because authors have cumulated studies in which
> >> samples were drawn from different levels" (p. 267).
> >>
> >> Can I can include these studies reporting effects from aggregated
> >> observations, and if so, are there specific procedures to do so? (I'm
> >> planning to use rma.mv in metafor, with cluster-robust variance
> >> estimates,
> >> using clubSandwich.)
> >>
> >> Many thanks!
> >> Fabian
> >>
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]