[R-meta] Transformation for ICC as outcome?

[Viechtbauer, Wolfgang (NP)]

Dear Andrew,

We did a meta-analysis of ICC(1) values here:

Nicolaï, S. P. A., Viechtbauer, W., Kruidenier, L. M., Candel, M. J. J. M., Prins, M. H. & Teijink, J. A. W. (2009). Reliability of treadmill testing in peripheral arterial disease: A meta-regression analysis. Journal of Vascular Surgery, 50(2), 322-329. https://doi.org/10.1016/j.jvs.2009.01.042 

For ICC(1) values, there is a version of Fisher's r-to-z transformation that is directly applicable.

However, note that there are many types of ICCs. An often given reference here is:

McGraw, K. O. & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. https://doi.org/10.1037/1082-989x.1.1.30 

It seems like you are dealing with ICCs that come from a multilevel model of the form:

lme(y ~ 1, random = ~ 1 | group)

and then computing ICC = var(group) / (var(group) + var(error)).

This is identical to the ICC(1) discussed above as long as var(group) > 0 and as long as the groups are all of equal size. So in that case, the r-to-z transformation for ICC(1) values is equally applicable. If group sizes differ, then this equivalence breaks down, but one can think of the ICC above as a sort of average ICC(1) (in McGraw & Wong, 1996, the k in the equation for the ICC(1) on page 35 is the number of subjects per group and if this differs, then we could plug in the average group size and end up with something that will typically be quite similar to the ICC above).

So, this is what I would go with in this scenario. So, using

yi = 1/2 * ln((1 + (n-1) * ICC) / (1 - ICC))
vi = n / (2*(n-1) * (p-2))

where n is the (average) group size and p is the number of groups.

Any other approach would require knowing the variance of the ICC values (or of some transformation thereof). I am not aware of any closed-form derivation thereof.

One difficulty (that we glossed over in the MA article) with the transformation above is that the yi values inherently estimate different parameters when n differs across studies. Back then, I did some convoluted transformation based on some simulations to 'equate' the transformed ICC(1) values calculated based on 3 assessments to those from 2 assessments. Instead, I would now include n as a predictor (and probably higher polynomial terms thereof, since the impact of n is non-linear) to account for this.

Best,
Wolfgang

-- 
Wolfgang Viechtbauer, PhD, Statistician | Department of Psychiatry and    
Neuropsychology | Maastricht University | PO Box 616 (VIJV1) | 6200 MD    
Maastricht, The Netherlands | +31(43)3884170 | https://www.wvbauer.com    

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Andrew McAleavey
>Sent: Friday, 10 June, 2022 11:24
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Transformation for ICC as outcome?
>
>Hi,
>
>tl;dr: What is the proper transformation to apply to ICC values as the
>outcome of a meta-analysis?
>
>More detail:
>
>I work in an area where it is substantively interesting to evaluate the
>intraclass correlation coefficient (ICC) from hierarchical models as an
>estimate of the relative impact of clustering level. Most frequently my
>field looks at patient outcomes in psychotherapy clustered within
>psychotherapists, but the same design is often used for group effects of
>group psychotherapy, clinic/center effects etc. The ICC, computed through
>variance components, is a conventional way to indicate whether these
>clustering levels make a difference in treatment outcome.
>
>From what I can tell, previous meta-analyses of these ICCs have not
>transformed the outcome variable at all, and the values tend to be quite
>low in published papers (.003 to .04 are common, slightly larger values
>occasionally). The lack of transformation seems strange to me, given that
>these ICCs are ratios of variance components. In fact, not only are these
>ICC values are bounded at (or close to) 0, but when estimates are this
>small, publication bias should be a huge factor due to non-convergence of
>ML estimates (primary studies would tend to simply omit this clustering
>factor if the model didn't converge or the effect was extremely small, so
>the file-drawer is essentially infinitely large). Without transforming the
>outcome, tests for publication bias would also be problematic, since there
>is no possibility for symmetry in a funnel plot (for example), right?
>
>Despite this, I haven't seen any examples of meta-analyses of ICC values in
>other fields that transform the ICC values first. So maybe this is not a
>big deal? Admittedly, it is hard to search for meta-analyses of ICC values
>as the outcome, and I haven't found that many outside my area at all -
>probably there are more I am not familiar with.
>
>My question is this:
>
>Does it make sense to transform ICC values prior to meta-analytic
>aggregation, and if so, what transformation makes the most sense?
>
>I've had logit and double-arcsine transformations recommended already since
>they apply for ratio/proportion outcomes. I am just not sure if I am
>missing some reason why ICC values should not be treated that way.
>
>Any advice or links would be appreciated!
>
>Best,
>Andrew McAleavey
>Helse Førde, Norway
>
>--
>Andrew McAleavey
>andrew.mcaleavey using gmail.com
>He / him