[R-sig-ME] metafor - Appropriate heterogeneity estimator when heterogeneity is extreme

Mon May 25 14:22:30 CEST 2015

Welcome to the wonderful world of heterogeneity (tau^2) estimators in meta-analysis. The question of which estimator to choose never fails to provide fun and entertainment for the entire family and I predict that it will do so for generations to come. Please quote me on that!

Now, to actually address your question. While in theory, the DL estimator is unbiased (under the assumptions of the random/mixed-effects model), there is some evidence out there that, in practice, the estimator underestimates the true amount of heterogeneity when tau^2 is in fact large, which seems to be especially an issue when meta-analyzing 2x2 table data (i.e., when meta-analyzing log odds/risk ratios and maybe also when meta-analyzing risk differences). And the REML estimator -- which cannot be proven to be unbiased, except under some special cases, but which has been studied quite extensively in simulation studies -- may possibly work better (on average) under such circumstances.

So, that may lead one to conclude that the results based on REML are closer to the truth for your particular case, but this is would be a rather big jump in conclusions. Bias is something that manifests itself *on average*, so in any particular case, we cannot know whether DL, REML, or any other of the dozens of tau^2 estimators out there is closest to the truth. So, while it may indeed be the case that the value you obtain from the DL estimator is also too small in your example, it could also be the one value that is most accurate.

So, in the end, that leaves you with some uncertainty as to what result you really should believe in. One thing that is certain though is that there is a lot of heterogeneity in these data, even when one would go with the DL estimator. Consider the results for 'e1' as an illustration. The log(RR) is -0.1973, which corresponds to a RR of about 0.82. With an estimate of tau^2 equal to 0.0805, we get a (very rough) 95% prediction interval for the true RR in any particular study (that is like an 'e1' study) with bounds:

round(exp(-0.1973 - 1.96*0.2837), 2)
round(exp(-0.1973 + 1.96*0.2837), 2)

that is, 0.47 to 1.43. So, in essence, while the true risk could be half as large in group 1 versus 2, it is not implausible that the true risk could also be almost 1.5 times larger in group 1 versus 2 (and this interval is still too narrow, as it ignores uncertainty in the estimate of the mean and the uncertainty in the estimate of tau^2 itself).

In the end, my recommendation would be to emphasize the large amount of heterogeneity, regardless of which estimator you go with, and to acknowledge that the exact amount is far from certain, due to the rather different values you obtain depending on which estimator you use. Formulate your conclusions accordingly (e.g., if you are inclined to say something about the average true log(RR) of 'e1' being significantly different from 0 or not based on the p-value being below .05 or not, do so cautiously!) and you've interpreted the data appropriately, at least as far as I am concerned (and based on the information given below).

Best,
Wolfgang

--    
Wolfgang Viechtbauer, Ph.D., Statistician    
Department of Psychiatry and Neuropsychology    
School for Mental Health and Neuroscience    
Faculty of Health, Medicine, and Life Sciences    
Maastricht University, P.O. Box 616 (VIJV1)    
6200 MD Maastricht, The Netherlands    
+31 (43) 388-4170 | http://www.wvbauer.com    

> -----Original Message-----
> From: R-sig-mixed-models [mailto:r-sig-mixed-models-bounces at r-
> project.org] On Behalf Of Theodore Lytras
> Sent: Monday, May 25, 2015 12:49
> To: r-sig-mixed-models at r-project.org
> Subject: [R-sig-ME] metafor - Appropriate heterogeneity estimator when
> heterogeneity is extreme
> 
> Hello everyone,
> 
> Sorry if this is not the right forum.
> I am undertaking a meta-regression using the metafor package, with six
> binary
> predictors and no intercept (I am only interested in the coefficients,
> not in
> the pooled effect). I have a sufficient number of studies (N=52), and the
> study effects (log Risk Ratios) show substantial left skew when plotted
> on a
> normal plot.
> 
> The problem is that I get very different estimates for tau^2 (and
> therefore
> quite different SEs for the regression coefficients) depending on whether
> I
> use the DerSimonian-Laird (DL) estimator or the REML estimator.
> In both cases heterogeneity is extreme (I^2>99, H^2>150), but t^2 is
> twice as
> big with REML (0.1901) than with DL (0.0805).
> 
> Could someone give me any clues as to which estimator may be more
> appropriate
> in such a situation?
> Also, what may be the cause of such greatly divergent estimates? Is it
> the
> heterogeneity? The non-normal distribution of study effects? Or something
> else?
> 
> Here's the output:
> 
> > rma.uni(yi=logRR, sei=SElogRR, mods=cbind(e1,e2,e3,e4,e5,e6),
> intercept=FALSE, method="DL", data=a)
> 
> Mixed-Effects Model (k = 52; tau^2 estimator: DL)
> 
> tau^2 (estimated amount of residual heterogeneity):     0.0805 (SE =
> 0.0387)
> tau (square root of estimated tau^2 value):             0.2837
> I^2 (residual heterogeneity / unaccounted variability): 99.46%
> H^2 (unaccounted variability / sampling variability):   185.55
> 
> Test for Residual Heterogeneity:
> QE(df = 46) = 8535.3517, p-val < .0001
> 
> Test of Moderators (coefficient(s) 1,2,3,4,5,6):
> QM(df = 6) = 314.3132, p-val < .0001
> 
> Model Results:
> 
>     estimate      se      zval    pval    ci.lb    ci.ub
> e1   -0.1973  0.0775   -2.5476  0.0108  -0.3491  -0.0455    *
> e2   -0.1679  0.0825   -2.0358  0.0418  -0.3296  -0.0063    *
> e3   -0.0413  0.0774   -0.5327  0.5942  -0.1930   0.1105
> e4   -0.1241  0.0990   -1.2539  0.2099  -0.3181   0.0699
> e5   -0.4916  0.1320   -3.7239  0.0002  -0.7503  -0.2329  ***
> e6   -1.6907  0.1064  -15.8893  <.0001  -1.8992  -1.4821  ***
> 
> ---
> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> 
> > rma.uni(yi=logRR, sei=SElogRR, mods=cbind(e1,e2,e3,e4,e5,e6),
> intercept=FALSE, method="REML", data=a)
> 
> Mixed-Effects Model (k = 52; tau^2 estimator: REML)
> 
> tau^2 (estimated amount of residual heterogeneity):     0.1901 (SE =
> 0.0411)
> tau (square root of estimated tau^2 value):             0.4360
> I^2 (residual heterogeneity / unaccounted variability): 99.77%
> H^2 (unaccounted variability / sampling variability):   437.00
> 
> Test for Residual Heterogeneity:
> QE(df = 46) = 8535.3517, p-val < .0001
> 
> Test of Moderators (coefficient(s) 1,2,3,4,5,6):
> QM(df = 6) = 142.3333, p-val < .0001
> 
> Model Results:
> 
>     estimate      se      zval    pval    ci.lb    ci.ub
> e1   -0.1987  0.1175   -1.6913  0.0908  -0.4289   0.0316    .
> e2   -0.1695  0.1243   -1.3638  0.1726  -0.4132   0.0741
> e3   -0.0491  0.1165   -0.4212  0.6736  -0.2774   0.1793
> e4   -0.1184  0.1502   -0.7880  0.4307  -0.4127   0.1760
> e5   -0.5140  0.1954   -2.6304  0.0085  -0.8970  -0.1310   **
> e6   -1.6950  0.1588  -10.6722  <.0001  -2.0063  -1.3837  ***
> 
> ---
> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> 
> 
> For completeness, I checked the other heterogeneity estimators available
> in
> metafor. Most are close to REML, expect Hunter-Schmidt (HS) which gives
> an
> even lower tau^2 than DL.
> 
>      method  tau2    I2     H2
> DL       DL 0.080 99.46 185.55
> HS       HS 0.036 98.81  84.07
> HE       HE 0.190 99.77 435.58
> ML       ML 0.167 99.74 384.58
> REML   REML 0.190 99.77 437.00
> EB       EB 0.190 99.77 437.01
> PM       PM 0.190 99.77 437.01
> SJ       SJ 0.195 99.78 447.29
> 
> Thank you very much,
> 
> Theodore Lytras
> 
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models