[R-meta] Metafor results tau^2 and R^2

[Viechtbauer, Wolfgang (SP)]

Hi All,

R^2 in the output of metafor is *not* R^2 from Higgins et al. (2002). It is in fact a (pseudo) coefficient of determination that goes back to Raudenbush (1994). It estimates how much of the (total) heterogeneity is accounted for by the moderator(s) included in the model. If the *residual* amount of heterogeneity (i.e., the unaccounted for heterogeneity) is 0 after including the moderator(s) in the model, then R^2 is going to be 100% (i.e., all of the heterogeneity has been accounted for). One would in fact expect then that the moderator (or set of moderators) is significant -- it would actually be a bit odd if a moderator accounts for all of the heterogeneity, but fails to be significant (although one could probably construct an example where this is the case). And reporting R^2 is definitely useful, although should be cautiously interpreted given that R^2 can be rather inaccurate when k is small (as discussed in López‐López et al., 2014).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org]
>On Behalf Of Dr. Gerta Rücker
>Sent: Saturday, 08 August, 2020 23:09
>To: Dustin Lee; r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Metafor results tau^2 and R^2
>
>Dear Dustin,
>
>The results you report show that in this analysis there was no
>between-study heterogeneity found at all. As explained in the message,
>all measures given are measures of heterogeneity, also R^2. You find all
>definitions in Higgins JP, Thompson SG. Quantifying heterogeneity in a
>meta-analysis. Stat Med. 2002;21(11):1539-1558. doi:10.1002/sim.1186.
>
>R^2 should not be confused with the coefficient of determination (which
>is also often denoted R^2). It is unusual to report the heterogeneity
>measure R^2 in a study report; most authors would report tau, tau^2 or I^2.
>
>See also Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue
>reliance on I(2) in assessing heterogeneity may mislead. /BMC Med Res
>Methodol/. 2008;8:79. Published 2008 Nov 27. doi:10.1186/1471-2288-8-79.
>
>Best,
>
>Gerta
>
>Am 08.08.2020 um 22:13 schrieb Dustin Lee:
>> Dear all,
>>
>> I am currently conducting a meta regression in which we are examining the
>> role of temporal effects (year of study) in the relationship between
>> organizational attitudes and job performance. Using a mixed-effects model
>> using ML estimation, our analyses have thus far produced results that do
>> not appear to be irregular.
>>
>> Our problem: With one relationship the analysis is showing the following:
>> tau^2 (estimated amount of residual heterogeneity):     0 (SE = 0.0152)
>> tau (square root of estimated tau^2 value):             0
>> I^2 (residual heterogeneity / unaccounted variability): 0.00%
>> H^2 (unaccounted variability / sampling variability):   1.00
>> R^2 (amount of heterogeneity accounted for):            100.00%
>>
>> However, the significance of the effect of 'year of study' is significant
>> along with the omnibus Q_M statistic. While I inherently understand this
>is
>> due to the way in which these values (R^2, tau^2, I^2, etc.) are
>calculated
>> and that it may be due to the smaller than ideal sample size (k =32) as
>> suggested by López‐López and colleagues (2014). I am unsure on how these
>> findings should be reported, particularly the 100% R^2 with the
>significant
>> predictor 'year of study' result.
>>
>> Thank you for any assistance you may be able to provide.
>>
>> All the best,
>>
>> Dustin
>>
>> Reference:
>> López‐López, J. A., Marín‐Martínez, F., Sánchez‐Meca, J., Van den
>> Noortgate, W., & Viechtbauer, W. (2014). Estimation of the predictive
>power
>> of the model in mixed‐effects meta‐regression: A simulation study.
>*British
>> Journal of Mathematical and Statistical Psychology*, *67*(1), 30-48.