[R-meta] I2 interpretation for Multilevel meta-analysis with moderators

[Ivan Jukic]

Hi Wolfgang, 

thank you for your prompt and helpful response.

Indeed, I don't know how I missed this from the equation you provided. It clearly looks like an R2-type metric (Raudenbush, 2009). I guess I was too much focused on I2. 

Of course, R2 will always be more meaningful, but I'm still trying to make sense out of this while looking at both metrics at the same time (i.e., I2 and pseudo R2). Considering correct interpretations of both metrics, I would expect them to complement each other, well, at least in case of models with moderators. However, I then thought more about this and came up with the reasoning below.

Let's consider my previous model with moderators having a total I2 of 68.35% (which can be decomposed to ~ 67.8, 0.51, and 31.7). From this, we can say that 68.35% of the remaining variance (that is not accounted for already by the moderators) is due to the sum of study and effect size level heterogeneity (the rest is due to sampling variance). Then, taking the model's pseudo R2 into account, we can now conclude how 49% of the between-study heterogeneity (so 49% of the 67.8% from the above) and 92% of the within-study heterogeneity (so 92% of the 0.51% from the above) are accounted for by the moderators. Is this correct? If so, I would argue that we need both to have a complete picture, though, I'm afraid I might be wrong here.

Cheers,
Ivan


From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
Sent: Wednesday, 13 October 2021 10:32 PM
To: Ivan Jukic <ivan.jukic using aut.ac.nz>; r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
Subject: RE: I2 interpretation for Multilevel meta-analysis with moderators 
 
The 48.97833 and 92.06504 are pseudo R^2 statistics, so they tell you that 49% of the between-study and 92% of the within-study heterogeneity are accounted for by the moderators. To me, that's more informative than saying something about how much of the unaccounted for variance is due to (the sum of) between- and within-study heterogeneity.

Best,
Wolfgang

>-----Original Message-----
>From: Ivan Jukic [mailto:ivan.jukic using aut.ac.nz]
>Sent: Wednesday, 13 October, 2021 4:32
>To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
>Subject: Re: I2 interpretation for Multilevel meta-analysis with moderators
>
>Dear Wolfgang,
>
>thank you for explaining this and providing an example - I really appreciate it.
>
>Indeed, in the posts I linked, you made it clear that I2 meaning is not very
>intuitive (or even meaningful) in models with moderators. However, I would like
>to use it because these moderators are the most important thing in my analysis. I
>was thinking about just looking at I2 without moderators but these analyses are
>less meaningful for the story that I'm trying to tell. I'm now considering
>reporting both.
>
>I get (77.747826 3.745411 18.506763) for res0 (i.e., model without moderators),
>and (67.8411969 0.5082705 31.6505327) for res1 (i.e., model with moderators).
>
>After reading your response I'm now unsure what the numbers obtained by 100 *
>pmax(0, (res0$sigma2 - res1$sigma2) / res0$sigma2) actually mean? I get (48.97833
>92.06504).
>
>In your response, you interpreted this as "how much of the between-study and
>within-study heterogeneity is accounted for by the moderators". Based on these
>numbers, it seems like a lot of between- and within-study heterogenity is
>accounted for by the moderators. However, I can't interpret this in the same way
>by just looking at I2 for res0 and res1 (81.49% and 68.35%, respectively, OR even
>individual heterogeneity components from each model)? I guess I'm still missing
>something here.
>
>Cheers,
>Ivan