[R-meta] Transformation for ICC as outcome?

[Viechtbauer, Wolfgang (NP)]

The r-to-z transformation for ICC(1) values involves n. So the parameter that is estimated by a transformed value is:

1/2 * ln((1 + (n_i-1) * rho_i) / (1 - rho_i)),

where rho_i is the true ICC in the ith study. When the n_i values differ across studies, the r-to-z transformed values are estimating different parameters (even if rho_i is constant across studies) and therefore are inherently heterogeneous. Adding n_i (and n_i^2, n_i^3, ...) as predictors can be used to account for this to some extent.

In contrast, this is not an issue for the r-to-z transformation for plain-old Pearson product-moment correlations, which is:

1/2 * ln((1 + r_i) / (1 - r_i)),

since this estimates

1/2 * ln((1 + rho_i) / (1 - rho_i)),

so if the rho_i values are homogeneous across studies, so are the transformed parameters.

Best,
Wolfgang

>-----Original Message-----
>From: Andrew McAleavey [mailto:andrew.mcaleavey using gmail.com]
>Sent: Friday, 10 June, 2022 14:42
>To: Viechtbauer, Wolfgang (NP)
>Cc: r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Transformation for ICC as outcome?
>
>Thank you for the fast response!
>
>Your guesses about my meaning seem very accurate, so I think I could use your
>suggestion for a quick re-analysis of existing studies, though you highlight a
>number of issues that make interpretation of these values very hard. Indeed,
>sample sizes vary widely in these studies (not to mention many other substantive
>variables), so a more complete accounting for the sources of variability would
>really be necessary to fully understand this I guess.
>
>I wonder what exactly you mean by "the yi values inherently estimate different
>parameters when n differs across studies," though. I have seen estimates from
>education where smaller group sizes tend to produce smaller ICC values, but is
>this what you mean? Is there some other effect to consider?
>
>Thanks again, this is very helpful to see.
>
>Andrew
>
>On Fri, Jun 10, 2022 at 12:56 PM Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>Dear Andrew,
>
>We did a meta-analysis of ICC(1) values here:
>
>Nicolaï, S. P. A., Viechtbauer, W., Kruidenier, L. M., Candel, M. J. J. M.,
>Prins, M. H. & Teijink, J. A. W. (2009). Reliability of treadmill testing in
>peripheral arterial disease: A meta-regression analysis. Journal of Vascular
>Surgery, 50(2), 322-329. https://doi.org/10.1016/j.jvs.2009.01.042
>
>For ICC(1) values, there is a version of Fisher's r-to-z transformation that is
>directly applicable.
>
>However, note that there are many types of ICCs. An often given reference here
>is:
>
>McGraw, K. O. & Wong, S. P. (1996). Forming inferences about some intraclass
>correlation coefficients. Psychological Methods, 1(1), 30-46.
>https://doi.org/10.1037/1082-989x.1.1.30
>
>It seems like you are dealing with ICCs that come from a multilevel model of the
>form:
>
>lme(y ~ 1, random = ~ 1 | group)
>
>and then computing ICC = var(group) / (var(group) + var(error)).
>
>This is identical to the ICC(1) discussed above as long as var(group) > 0 and as
>long as the groups are all of equal size. So in that case, the r-to-z
>transformation for ICC(1) values is equally applicable. If group sizes differ,
>then this equivalence breaks down, but one can think of the ICC above as a sort
>of average ICC(1) (in McGraw & Wong, 1996, the k in the equation for the ICC(1)
>on page 35 is the number of subjects per group and if this differs, then we could
>plug in the average group size and end up with something that will typically be
>quite similar to the ICC above).
>
>So, this is what I would go with in this scenario. So, using
>
>yi = 1/2 * ln((1 + (n-1) * ICC) / (1 - ICC))
>vi = n / (2*(n-1) * (p-2))
>
>where n is the (average) group size and p is the number of groups.
>
>Any other approach would require knowing the variance of the ICC values (or of
>some transformation thereof). I am not aware of any closed-form derivation
>thereof.
>
>One difficulty (that we glossed over in the MA article) with the transformation
>above is that the yi values inherently estimate different parameters when n
>differs across studies. Back then, I did some convoluted transformation based on
>some simulations to 'equate' the transformed ICC(1) values calculated based on 3
>assessments to those from 2 assessments. Instead, I would now include n as a
>predictor (and probably higher polynomial terms thereof, since the impact of n is
>non-linear) to account for this.
>
>Best,
>Wolfgang
>
>--
>Wolfgang Viechtbauer, PhD, Statistician | Department of Psychiatry and
>Neuropsychology | Maastricht University | PO Box 616 (VIJV1) | 6200 MD
>Maastricht, The Netherlands | +31(43)3884170 | https://www.wvbauer.com
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Andrew McAleavey
>>Sent: Friday, 10 June, 2022 11:24
>>To: r-sig-meta-analysis using r-project.org
>>Subject: [R-meta] Transformation for ICC as outcome?
>>
>>Hi,
>>
>>tl;dr: What is the proper transformation to apply to ICC values as the
>>outcome of a meta-analysis?
>>
>>More detail:
>>
>>I work in an area where it is substantively interesting to evaluate the
>>intraclass correlation coefficient (ICC) from hierarchical models as an
>>estimate of the relative impact of clustering level. Most frequently my
>>field looks at patient outcomes in psychotherapy clustered within
>>psychotherapists, but the same design is often used for group effects of
>>group psychotherapy, clinic/center effects etc. The ICC, computed through
>>variance components, is a conventional way to indicate whether these
>>clustering levels make a difference in treatment outcome.
>>
>>From what I can tell, previous meta-analyses of these ICCs have not
>>transformed the outcome variable at all, and the values tend to be quite
>>low in published papers (.003 to .04 are common, slightly larger values
>>occasionally). The lack of transformation seems strange to me, given that
>>these ICCs are ratios of variance components. In fact, not only are these
>>ICC values are bounded at (or close to) 0, but when estimates are this
>>small, publication bias should be a huge factor due to non-convergence of
>>ML estimates (primary studies would tend to simply omit this clustering
>>factor if the model didn't converge or the effect was extremely small, so
>>the file-drawer is essentially infinitely large). Without transforming the
>>outcome, tests for publication bias would also be problematic, since there
>>is no possibility for symmetry in a funnel plot (for example), right?
>>
>>Despite this, I haven't seen any examples of meta-analyses of ICC values in
>>other fields that transform the ICC values first. So maybe this is not a
>>big deal? Admittedly, it is hard to search for meta-analyses of ICC values
>>as the outcome, and I haven't found that many outside my area at all -
>>probably there are more I am not familiar with.
>>
>>My question is this:
>>
>>Does it make sense to transform ICC values prior to meta-analytic
>>aggregation, and if so, what transformation makes the most sense?
>>
>>I've had logit and double-arcsine transformations recommended already since
>>they apply for ratio/proportion outcomes. I am just not sure if I am
>>missing some reason why ICC values should not be treated that way.
>>
>>Any advice or links would be appreciated!
>>
>>Best,
>>Andrew McAleavey
>>Helse Førde, Norway
>>
>>--
>>Andrew McAleavey
>>andrew.mcaleavey using gmail.com
>>He / him