[R-meta] Transformation for ICC as outcome?

[Andrew McAleavey]

Thank you for the fast response!

Your guesses about my meaning seem very accurate, so I think I could use
your suggestion for a quick re-analysis of existing studies, though you
highlight a number of issues that make interpretation of these values very
hard. Indeed, sample sizes vary widely in these studies (not to mention
many other substantive variables), so a more complete accounting for the
sources of variability would really be necessary to fully understand this I
guess.

I wonder what exactly you mean by "the yi values inherently estimate
different parameters when n differs across studies," though. I have seen
estimates from education where smaller group sizes tend to produce smaller
ICC values, but is this what you mean? Is there some other effect to
consider?

Thanks again, this is very helpful to see.

Andrew

On Fri, Jun 10, 2022 at 12:56 PM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Dear Andrew,
>
> We did a meta-analysis of ICC(1) values here:
>
> Nicolaï, S. P. A., Viechtbauer, W., Kruidenier, L. M., Candel, M. J. J.
> M., Prins, M. H. & Teijink, J. A. W. (2009). Reliability of treadmill
> testing in peripheral arterial disease: A meta-regression analysis. Journal
> of Vascular Surgery, 50(2), 322-329.
> https://doi.org/10.1016/j.jvs.2009.01.042
>
> For ICC(1) values, there is a version of Fisher's r-to-z transformation
> that is directly applicable.
>
> However, note that there are many types of ICCs. An often given reference
> here is:
>
> McGraw, K. O. & Wong, S. P. (1996). Forming inferences about some
> intraclass correlation coefficients. Psychological Methods, 1(1), 30-46.
> https://doi.org/10.1037/1082-989x.1.1.30
>
> It seems like you are dealing with ICCs that come from a multilevel model
> of the form:
>
> lme(y ~ 1, random = ~ 1 | group)
>
> and then computing ICC = var(group) / (var(group) + var(error)).
>
> This is identical to the ICC(1) discussed above as long as var(group) > 0
> and as long as the groups are all of equal size. So in that case, the
> r-to-z transformation for ICC(1) values is equally applicable. If group
> sizes differ, then this equivalence breaks down, but one can think of the
> ICC above as a sort of average ICC(1) (in McGraw & Wong, 1996, the k in the
> equation for the ICC(1) on page 35 is the number of subjects per group and
> if this differs, then we could plug in the average group size and end up
> with something that will typically be quite similar to the ICC above).
>
> So, this is what I would go with in this scenario. So, using
>
> yi = 1/2 * ln((1 + (n-1) * ICC) / (1 - ICC))
> vi = n / (2*(n-1) * (p-2))
>
> where n is the (average) group size and p is the number of groups.
>
> Any other approach would require knowing the variance of the ICC values
> (or of some transformation thereof). I am not aware of any closed-form
> derivation thereof.
>
> One difficulty (that we glossed over in the MA article) with the
> transformation above is that the yi values inherently estimate different
> parameters when n differs across studies. Back then, I did some convoluted
> transformation based on some simulations to 'equate' the transformed ICC(1)
> values calculated based on 3 assessments to those from 2 assessments.
> Instead, I would now include n as a predictor (and probably higher
> polynomial terms thereof, since the impact of n is non-linear) to account
> for this.
>
> Best,
> Wolfgang
>
> --
> Wolfgang Viechtbauer, PhD, Statistician | Department of Psychiatry and
> Neuropsychology | Maastricht University | PO Box 616 (VIJV1) | 6200 MD
> Maastricht, The Netherlands | +31(43)3884170 | https://www.wvbauer.com
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Andrew McAleavey
> >Sent: Friday, 10 June, 2022 11:24
> >To: r-sig-meta-analysis using r-project.org
> >Subject: [R-meta] Transformation for ICC as outcome?
> >
> >Hi,
> >
> >tl;dr: What is the proper transformation to apply to ICC values as the
> >outcome of a meta-analysis?
> >
> >More detail:
> >
> >I work in an area where it is substantively interesting to evaluate the
> >intraclass correlation coefficient (ICC) from hierarchical models as an
> >estimate of the relative impact of clustering level. Most frequently my
> >field looks at patient outcomes in psychotherapy clustered within
> >psychotherapists, but the same design is often used for group effects of
> >group psychotherapy, clinic/center effects etc. The ICC, computed through
> >variance components, is a conventional way to indicate whether these
> >clustering levels make a difference in treatment outcome.
> >
> >From what I can tell, previous meta-analyses of these ICCs have not
> >transformed the outcome variable at all, and the values tend to be quite
> >low in published papers (.003 to .04 are common, slightly larger values
> >occasionally). The lack of transformation seems strange to me, given that
> >these ICCs are ratios of variance components. In fact, not only are these
> >ICC values are bounded at (or close to) 0, but when estimates are this
> >small, publication bias should be a huge factor due to non-convergence of
> >ML estimates (primary studies would tend to simply omit this clustering
> >factor if the model didn't converge or the effect was extremely small, so
> >the file-drawer is essentially infinitely large). Without transforming the
> >outcome, tests for publication bias would also be problematic, since there
> >is no possibility for symmetry in a funnel plot (for example), right?
> >
> >Despite this, I haven't seen any examples of meta-analyses of ICC values
> in
> >other fields that transform the ICC values first. So maybe this is not a
> >big deal? Admittedly, it is hard to search for meta-analyses of ICC values
> >as the outcome, and I haven't found that many outside my area at all -
> >probably there are more I am not familiar with.
> >
> >My question is this:
> >
> >Does it make sense to transform ICC values prior to meta-analytic
> >aggregation, and if so, what transformation makes the most sense?
> >
> >I've had logit and double-arcsine transformations recommended already
> since
> >they apply for ratio/proportion outcomes. I am just not sure if I am
> >missing some reason why ICC values should not be treated that way.
> >
> >Any advice or links would be appreciated!
> >
> >Best,
> >Andrew McAleavey
> >Helse Førde, Norway
> >
> >--
> >Andrew McAleavey
> >andrew.mcaleavey using gmail.com
> >He / him
>


-- 
Andrew McAleavey
andrew.mcaleavey using gmail.com
He / him

	[[alternative HTML version deleted]]