[R-meta] Combining studies reporting effects at different level of analysis/aggregation

[F S]

Dear James,

Thanks a lot for your reply -- I had suspected that it may not be possible
to recover the individual-level correlations and separate meta-analyses
would be required (see Ostroff & Harrison and others). Quel dommage...

All best,
Fabian


On Sun, Oct 28, 2018 at 9:09 PM James Pustejovsky <jepusto using gmail.com> wrote:

> Fabian,
>
> As far as I can see, it is not generally possible to recover
> individual-level correlations from group averages. As Ostroff & Harrison
> (among others) so, the overall individual-level correlation is a weighted
> average of the within-group correlation and the correlation of group
> averages. The correlation of group averages is not informative about the
> within-group correlation (the two could even be of opposite signs, as in
> Simpson's paradox), and so without outside information or very strong
> assumptions, I see no way to figure out the within-group correlation
> contribution to the overall individual-level correlation.
>
> As an aside, even though in practice most meta-analysts probably focus
> most on the overall individual-level correlation, I would argue that we
> should give greater consideration to parsing out within-group correlations
> from group-average correlations. Because the two parameters may be very
> different---and because they may have quite distinct substantive
> interpretations---I think that ideally we should be to try and conduct
> syntheses that also keep them distinct (e.g., conducting separate
> meta-analyses of the two types of correlations).  Of course this may be
> difficult or impossible in practice due to the limitations of primary study
> reporting. C'est la vie d'un meta-analyste...
>
> James
>
> On Sun, Oct 28, 2018 at 1:27 PM F S <crpt.fs using gmail.com> wrote:
>
>> Dear Wolfgang,
>>
>> Thanks again for your very helpful advice! I've been digging into a few
>> other resources that are in line with your recommendations (e.g., Hedges,
>> 2007, 2009) and am happy with the correction approach we discussed.
>> However, this approach covers specifically SMDs – which was indeed the
>> scenario in my meta-analysis that prompted me to consult the mailing list.
>> I now wonder, more out of general interest, how one would go about
>> correcting results from aggregated/averaged data if the effect size in
>> question is a correlation coefficient.
>>
>> The potential for bias from aggregation seems to be quite substantial in
>> the domain of correlations. For instance, Brand and Bradley (2012)
>> demonstrated that correlations of averages are upwardly biased relative to
>> individual-level correlations by up to 76%. However, I didn't come across
>> any procedures for correcting inflated correlations of group means for
>> inclusion in a meta-analysis. Nickerson (1995) presented a formula for the
>> relationship between correlations of group averages and average
>> within-group correlations, but what a meta-analyst would want to compute
>> is
>> not the within-group correlation, but an estimate of the individual-level
>> correlation across groups. To complicate matters further, in addition to
>> inflating the magnitude of the effect size by correlating group means
>> instead of individual observations, there is the additional possibility
>> that aggregation distorts the direction of the individual-level effect
>> (e.g., Ostroff & Harrison, 1999).
>>
>> Do you have any recommendations for how one would correct correlations of
>> group means to permit inclusion in a meta-analysis of individual-level
>> correlations? Is this possible at all?
>>
>> Many thanks,
>> Fabian
>>
>> ---
>> References:
>> Brand, A., & Bradley, M. T. (2012). More voodoo vorrelations: When
>> average-based measures inflate correlations. The Journal of General
>> Psychology, 2012, 139, 260-272.
>> Hedges, L. V. (2007). Effect sizes in cluster randomized designs. Journal
>> of Educational and Behavioral Statistics 32, 341-370.
>> Hedges, L. V. (2009). Effect sizes in nested designs. In H. Cooper, L.
>> V.Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis
>> (2nd
>> ed., pp. 337-355). New York: RussellSage.
>> Nickerson, C. A. E. (1995). Does willingness to pay reflect the purchase
>> of
>> moral satisfaction? A reconsideration of Kahneman and Knetsch. Journal of
>> Environmental Economics and Management, 28, 126-133.
>> Ostroff C, Harrison DA. (1999). Meta-analysis, level of analysis, and best
>> estimates of population correlations: Cautions for interpreting
>> meta-analytic results in organizational behavior. Journal of Applied
>> Psychology, 84, 260-270.
>>
>>
>>
>>
>>
>>
>> On Sun, Oct 14, 2018 at 2:30 PM F S <crpt.fs using gmail.com> wrote:
>>
>> > Dear Wolfgang,
>> > Thanks for clarifying -- I will attempt this approach then, and also
>> > include study type as a moderator, as per your recommendation.
>> > All the best,
>> > Fabian
>> >
>> > On Thu, Oct 11, 2018 at 1:20 PM Viechtbauer, Wolfgang (SP) <
>> > wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >
>> >> Please always cc the mailing list when replying.
>> >>
>> >> Yes, you could also 'guestimate' the ICC and use that (and then do a
>> >> sensitivity analysis). Even if you do the correction, I would still
>> >> recommend to include study type as a moderator in the analyses.
>> >>
>> >> Best,
>> >> Wolfgang
>> >>
>> >> -----Original Message-----
>> >> From: F S [mailto:crpt.fs using gmail.com]
>> >> Sent: Thursday, 11 October, 2018 18:47
>> >> To: Viechtbauer, Wolfgang (SP)
>> >> Subject: Re: [R-meta] Combining studies reporting effects at different
>> >> level of analysis/aggregation
>> >>
>> >> Hello Wolfgang,
>> >>
>> >> Thank you for your helpful answer. I'm afraid none of the studies in
>> >> question report the ICC, so I guess a precise correction for the
>> inflated d
>> >> won't be possible. However, would it be sensible to instead impute a
>> value
>> >> for rho and perform the adjustment for the design effect using that
>> value?
>> >> Ideally, one would impute ICC values lifted from studies with a similar
>> >> type of aggregation and similar measures, but I suppose one could also
>> >> perform the correction for a range of plausible values of rho and
>> evaluate
>> >> the impact on the overall results via sensitivity analysis. What do you
>> >> think?
>> >>
>> >> Thank you very much,
>> >> Fabian
>> >>
>> >> On Fri, Oct 5, 2018 at 1:17 PM Viechtbauer, Wolfgang (SP) <
>> >> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >> Hi Fabian,
>> >>
>> >> I don't think you have received any responses to your question so far,
>> so
>> >> let me take a stab here.
>> >>
>> >> You did not say what kind of effect size / outcome measure you want to
>> >> use for your meta-analysis, but if it something like a standardized
>> mean
>> >> difference ('d-values'), then what you describe is definitely an
>> issue. The
>> >> means (i.e., the averaged individual responses within groups) will
>> have a
>> >> lower variance than the responses from individuals, leading to higher
>> >> d-values in studies reporting statistics based on group-level means.
>> That
>> >> makes d-values from the two types of studies pretty much
>> non-comparable. At
>> >> the very least, you should include study type as a moderator in all of
>> the
>> >> analyses.
>> >>
>> >> If you know the ICC of the responses within groups, then one could
>> >> correct for the inflation of the d-values based on the 'variance
>> inflation
>> >> factor' or 'design effect'. In essence, d-values from 'group studies'
>> are
>> >> then adjusted by the multiplicative factor
>> >>
>> >> sqrt((1+(n-1)*rho)/n),
>> >>
>> >> where n is the (average) group size and rho is the ICC. That should
>> make
>> >> the d-values from the two types of studies more directly comparable.
>> The
>> >> sampling variance of a d-value from a 'group study' also needs to be
>> >> adjusted based on the square of the multiplicative factor (this
>> ignores the
>> >> uncertainty in the estimated value of the ICC, but ignoring sources of
>> >> uncertainty when estimating sampling variances happens all the time).
>> >>
>> >> Best,
>> >> Wolfgang
>> >>
>> >> -----Original Message-----
>> >> From: R-sig-meta-analysis [mailto:
>> >> r-sig-meta-analysis-bounces using r-project.org] On Behalf Of F S
>> >> Sent: Tuesday, 18 September, 2018 20:47
>> >> To: r-sig-meta-analysis using r-project.org
>> >> Subject: [R-meta] Combining studies reporting effects at different
>> level
>> >> of analysis/aggregation
>> >>
>> >> I am currently working on a meta-analysis in the social sciences. All
>> >> studies measured the relevant outcome at the level of participants,
>> but a
>> >> few studies aggregated at a higher level of analysis (e.g., groups)
>> before
>> >> statistics were computed. Can these studies be meta-analyzed together?
>> >>
>> >> More detail: The relevant outcome is a continuous measure, assessed at
>> the
>> >> level of individual participants. The majority of studies report
>> >> statistical effects computed at the level of participants. However, in
>> a
>> >> number of studies, random assignment occurred not at the participant
>> >> level,
>> >> but at the level of groups (e.g., dyads, 3-person groups, classrooms).
>> >> Although each of these studies did assess the outcome at the
>> participant
>> >> level, just like the other studies, statistical effects are computed at
>> >> the
>> >> group level. As such, they are different from cluster-randomized
>> studies,
>> >> in which randomization occurs at the group level but results are
>> reported
>> >> at the individual level. By contrast, the studies in question averaged
>> >> individual responses within groups before computing effects with group
>> as
>> >> the unit of analysis.
>> >>
>> >> I'm not sure I can include these studies in my meta-analysis, but could
>> >> not
>> >> find much work on this question. Ostroff and Harrison (1999) focused
>> >> specifically on correlations computed at different levels of analysis,
>> and
>> >> they make a strong case against combining ES from such studies: "the
>> >> obtained meta-analytic ρ̂  may not be interpretable as an estimate of
>> any
>> >> population parameter because authors have cumulated studies in which
>> >> samples were drawn from different levels" (p. 267).
>> >>
>> >> Can I can include these studies reporting effects from aggregated
>> >> observations, and if so, are there specific procedures to do so? (I'm
>> >> planning to use rma.mv in metafor, with cluster-robust variance
>> >> estimates,
>> >> using clubSandwich.)
>> >>
>> >> Many thanks!
>> >> Fabian
>> >>
>> >
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

	[[alternative HTML version deleted]]