[R-meta] metafor::rma-function: Statistically significant interaction, but increased tau2 – and how to get the slope from the output
Michael Dewey
||@t@ @end|ng |rom dewey@myzen@co@uk
Fri Aug 2 15:54:21 CEST 2019
Dear Sabrina
Just a side issue but the journals I review for usually take the line
that deviating from the protocol is fine as long as you say you have
done so and why you did it so readers can tell whether it was just
data-driven.
Michael
On 02/08/2019 14:25, Sabrina Mai Nielsen wrote:
> Thanks a lot for very fast response!
>
> 1) Ok, I am happy to hear that it is a known issue and not necessarily caused by errors in my coding.
> 2) Perfect, that makes sense. Thanks a lot for making that clear for me.
> 3) Thanks a lot for the suggestions - I will try those models out! For my study, however, I have already protocolized the model I am using, so I may have to stick with that.
> - For the model that allows tau^2 to be different for every level of id, I do get the same as with the 'subset' model, as you said - cool!
> - The multilevel model adding random effects for meta-analyses was definitely among our considerations, however, we found it rather complex (e.g. by resulting in two variance components, sigma^2.1 and sigma^2.2) and we ended up choosing fixed effects for meta-analyses.
> - The profile plots are peaking nicely for the two first plots (σ^2 and tau1^2), but not for the last two plots.
>
> Thanks again
>
> Best,
> Sabrina
>
> -----Oprindelig meddelelse-----
> Fra: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Sendt: 1. august 2019 22:06
> Til: Sabrina Mai Nielsen <sabrina.mai.nielsen using regionh.dk>; r-sig-meta-analysis using r-project.org
> Emne: RE: metafor::rma-function: Statistically significant interaction, but increased tau2 – and how to get the slope from the output
>
> Hi Sabrina,
>
> 1) While conceptually, tau^2 should never increase when adding moderators to a meta-regression model, this can happen in practice (as in your case). It's a bit weird when this happens, especially when the moderators added (or interactions in your case) are actually significant. I can't tell you why this happens in your case specifically.
>
> 2) The slope for id 8 is 0.0114 + (-0.0718). When you select only the studies for id 8, then you allow tau^2 to be different in that subset (and hence different from tau^2 for the entire collection) and that can cause such a discrepancy. And it is just a coincidence that -0.0718 is closer to -0.0940. And yes, it can happen that the slopes in the subsets are not significant, but that some of the *differences* between the slope for id 1 and the slopes for the other ids are.
>
> You can also fit a model where you allow tau^2 to be different for every level of id. That would be:
>
> d$row <- 1:nrow(d)
> rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP * id, random = ~ factor(id) | row, struct="DIAG", data=d, btt=21:38, verbose=TRUE)
>
> It could take a while to fit that model, since 19 different values of tau^2 will need to be estimated (verbose=TRUE will keep you updated on the progress). Once the model converges, you should find that the results are now (essentially) identical to what you find if you fit the 'subset' models, such as:
>
> rma(logOR, logOR.var, mods= ~ CF1_Women..IMP, data=d, subset=id==8)
>
> 3) You might want to take another approach here, since the data have a multilevel structure (with studies nested within meta-analyses). A multilevel model adding random effects for meta-analyses and studies within meta-analyses can be fitted with:
>
> rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP, random = ~ 1 | id/row, data=d)
>
> This model assumes that the slope for CF1_Women..IMP is identical across meta-analyses (so it is analogous to the model without interactions). To examine if the slope varies across meta-analyses, you could use a model with random slopes:
>
> rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP, random = ~ CF1_Women..IMP | id, struct="GEN", data=d)
>
> This fits a model with random intercepts and slopes for meta-analyses (with a var-cov structure that allows intercepts and slopes to be correlated). Note: The "GEN" structure is currently undocumented, as I still need to make a few of the method functions work with it. anova() for a model comparison should work though.
>
> You might even want to try:
>
> res <- rma.mv(logOR, logOR.var, mods = ~ CF1_Women..IMP, random = list(~ CF1_Women..IMP | id, ~ 1 | row), struct="GEN", data=d) res
>
> which also adds random effects for studies within meta-analyses. I am not entirely sure right now if this is an overparameterized model, so you would want to profile all variance/correlation components of this model with:
>
> profile(res)
>
> This could take a while, so be patient. All profile plots should have a peak at the estimate. If one or more of the profile likelihoods are flat around the estimate, then you know that the model is overparameterized.
>
> If you pursue these models, I would be interested in some feedback as to how this worked out (and how those profile plots look).
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Sabrina Mai Nielsen
> Sent: Thursday, 01 August, 2019 20:52
> To: r-sig-meta-analysis using r-project.org
> Subject: [R-meta] metafor::rma-function: Statistically significant interaction, but increased tau2 – and how to get the slope from the output
>
> I am having troubles making sense of a REML-based meta-regression with an interaction between a factor and numeric variable, fitted with rma.
>
> In my study, I have included RCTs from 19 meta-analyses to investigate the association between %women in the RCTs and the effect sizes (i.e. logOR).
> The slope for %women, without interaction, is not significant (see below, M_REML0, p=0.6858), and I am now investigating the interaction between %women (CF1_Women..IMP) and the ID for the meta-analysis (i.e. if the slope for %women varies in the different meta-analyses).
>
> 1) The interaction seem significant when testing for moderators, but the between-trial variation, tau^2, does not decrease as expected. Why is that?
> (When I try method="ML" instead of REML, then tau^2 does decrease as expected; see anova(M0,M1) below)
>
> M_REML0 (without interaction), tau^2 = 0.2076
> M_REML1 (with an interaction), tau^2 = 0.2127, Test of Moderators (interaction terms) p-val = 0.0044 (see full outputs below)
>
> 2) I am in doubt how to get the estimate for the slopes – e.g. for meta-analysis with ID 8.
>
> estimate se zval pval ci.lb ci.ub
> intrcpt -0.3567 1.0739 -0.3322 0.7398 -2.4616 1.7481
> CF1_Women..IMP 0.0114 0.0139 0.8244 0.4097 -0.0157 0.0386
> (...)
> CF1_Women..IMP:id8 -0.0718 0.0214 -3.3572 0.0008 -0.1137 -0.0299 ***
> (see full output below)
>
> I would think it would be calculated as CF1_Women..IMP + CF1_Women..IMP:id8, i.e. 0.0114 +(-0.0718) = -0.0604 (with corresponding 95%CI of -0.1294 to 0.0087). Is that correct?
> Or is the slope for ID 8 simply the estimate reported for CF1_Women..IMP:id8, i.e. (-0.0718, 95%CI -0.1137 to -0.0299)?
> I am asking because when I do a meta-regression with only RCTs for ID 8, I get -0.0940 (95%CI -0.1864 to -0.0016; output not shown), which is more similar to the CF1_Women..IMP:id8 estimate.
>
> Also, when I calculate the slopes for all the meta-analysis IDs, none of them are significantly different from 0, despite the interaction term is significant. I guess that is possible, but it makes me doubt the way I calculated the slopes.
>
> Thanks a lot in advance! Your help is very appreciated!
> Best,
> Sabrina
>
> =======A FEW RELEVANT OUTPUTS=======
>> M_REML0<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP + id,
>> method="REML", data=d, btt=2) M_REML1<-rma(logOR, logOR.var, mods=
>> ~CF1_Women..IMP * id, method="REML", data=d, btt=c(21:38))
>> M_REML0
>
> Mixed-Effects Model (k = 212; tau^2 estimator: REML)
>
> tau^2 (estimated amount of residual heterogeneity): 0.2076 (SE = 0.0357)
> tau (square root of estimated tau^2 value): 0.4556
> I^2 (residual heterogeneity / unaccounted variability): 66.60%
> H^2 (unaccounted variability / sampling variability): 2.99
> R^2 (amount of heterogeneity accounted for): 63.70%
>
> Test for Residual Heterogeneity:
> QE(df = 192) = 545.9074, p-val < .0001
>
> Test of Moderators (coefficient 2):
> QM(df = 1) = 0.1637, p-val = 0.6858
>
> Model Results:
>
> estimate se zval pval ci.lb ci.ub
> intrcpt 0.4156 0.3105 1.3383 0.1808 -0.1931 1.0242
> CF1_Women..IMP 0.0013 0.0032 0.4046 0.6858 -0.0049 0.0075
> id2 0.0672 0.3965 0.1695 0.8654 -0.7099 0.8443
> id3 0.5059 0.2521 2.0066 0.0448 0.0117 1.0000 *
> id4 0.1753 0.2739 0.6402 0.5221 -0.3615 0.7121
> id5 0.2610 0.2558 1.0201 0.3077 -0.2404 0.7624
> id6 0.0351 0.2763 0.1272 0.8988 -0.5065 0.5768
> id7 0.8416 0.3464 2.4295 0.0151 0.1626 1.5205 *
> id8 0.1775 0.2723 0.6519 0.5145 -0.3562 0.7112
> id9 -0.3137 0.2684 -1.1690 0.2424 -0.8398 0.2123
> id10 -0.8378 0.2723 -3.0765 0.0021 -1.3715 -0.3040 **
> id11 0.2878 0.2106 1.3665 0.1718 -0.1250 0.7006
> id12 0.1932 0.3449 0.5603 0.5753 -0.4827 0.8692
> id13 0.1538 0.2959 0.5199 0.6031 -0.4261 0.7337
> id14 -1.0020 0.3432 -2.9195 0.0035 -1.6747 -0.3293 **
> id15 -0.6162 0.2739 -2.2497 0.0245 -1.1531 -0.0794 *
> id16 -1.5704 0.3394 -4.6269 <.0001 -2.2356 -0.9052 ***
> id17 -1.4576 0.2942 -4.9552 <.0001 -2.0341 -0.8811 ***
> id18 -0.0457 0.4967 -0.0921 0.9266 -1.0193 0.9278
> id19 -0.3440 0.3283 -1.0479 0.2947 -0.9874 0.2994
>
> ---
> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
>> M_REML1
>
> Mixed-Effects Model (k = 212; tau^2 estimator: REML)
>
> tau^2 (estimated amount of residual heterogeneity): 0.2127 (SE = 0.0378)
> tau (square root of estimated tau^2 value): 0.4612
> I^2 (residual heterogeneity / unaccounted variability): 67.50%
> H^2 (unaccounted variability / sampling variability): 3.08
> R^2 (amount of heterogeneity accounted for): 62.80%
>
> Test for Residual Heterogeneity:
> QE(df = 174) = 488.8555, p-val < .0001
>
> Test of Moderators (coefficients 21:38):
> QM(df = 18) = 37.5693, p-val = 0.0044
>
> Model Results:
>
> estimate se zval pval ci.lb ci.ub
> intrcpt -0.3567 1.0739 -0.3322 0.7398 -2.4616 1.7481
> CF1_Women..IMP 0.0114 0.0139 0.8244 0.4097 -0.0157 0.0386
> id2 8.6017 8.6515 0.9942 0.3201 -8.3549 25.5583
> id3 0.7355 1.3494 0.5451 0.5857 -1.9093 3.3804
> id4 0.2098 1.2889 0.1628 0.8707 -2.3164 2.7360
> id5 1.7079 1.1849 1.4414 0.1495 -0.6144 4.0302
> id6 -3.6331 2.2106 -1.6435 0.1003 -7.9658 0.6995
> id7 -5.4865 2.7547 -1.9917 0.0464 -10.8857 -0.0874 *
> id8 5.2745 1.5782 3.3421 0.0008 2.1812 8.3677 ***
> id9 0.4174 1.2215 0.3417 0.7326 -1.9766 2.8114
> id10 -0.1295 1.2015 -0.1078 0.9142 -2.4844 2.2253
> id11 0.2269 1.7098 0.1327 0.8944 -3.1242 3.5779
> id12 -0.7519 2.3333 -0.3222 0.7473 -5.3250 3.8212
> id13 -0.0701 3.4723 -0.0202 0.9839 -6.8758 6.7355
> id14 -0.1842 2.0870 -0.0882 0.9297 -4.2747 3.9063
> id15 0.9002 1.3974 0.6442 0.5194 -1.8386 3.6389
> id16 6.4300 7.6265 0.8431 0.3992 -8.5176 21.3776
> id17 0.7142 2.1172 0.3373 0.7359 -3.4353 4.8638
> id18 -0.4641 1.5171 -0.3059 0.7597 -3.4376 2.5095
> id19 1.2194 4.9584 0.2459 0.8057 -8.4989 10.9376
> CF1_Women..IMP:id2 -0.1243 0.1269 -0.9795 0.3274 -0.3730 0.1244
> CF1_Women..IMP:id3 -0.0026 0.0177 -0.1480 0.8824 -0.0374 0.0322
> CF1_Women..IMP:id4 0.0012 0.0174 0.0672 0.9464 -0.0329 0.0353
> CF1_Women..IMP:id5 -0.0207 0.0157 -1.3162 0.1881 -0.0515 0.0101
> CF1_Women..IMP:id6 0.0533 0.0308 1.7332 0.0831 -0.0070 0.1136 .
> CF1_Women..IMP:id7 0.0690 0.0313 2.2031 0.0276 0.0076 0.1304 *
> CF1_Women..IMP:id8 -0.0718 0.0214 -3.3572 0.0008 -0.1137 -0.0299 ***
> CF1_Women..IMP:id9 -0.0095 0.0162 -0.5858 0.5580 -0.0413 0.0223
> CF1_Women..IMP:id10 -0.0092 0.0156 -0.5907 0.5547 -0.0399 0.0214
> CF1_Women..IMP:id11 0.0004 0.0217 0.0167 0.9867 -0.0422 0.0430
> CF1_Women..IMP:id12 0.0112 0.0290 0.3852 0.7001 -0.0456 0.0679
> CF1_Women..IMP:id13 0.0042 0.0496 0.0856 0.9318 -0.0929 0.1014
> CF1_Women..IMP:id14 -0.0108 0.0281 -0.3837 0.7012 -0.0658 0.0443
> CF1_Women..IMP:id15 -0.0207 0.0186 -1.1133 0.2656 -0.0570 0.0157
> CF1_Women..IMP:id16 -0.0834 0.0777 -1.0733 0.2831 -0.2356 0.0689
> CF1_Women..IMP:id17 -0.0252 0.0239 -1.0543 0.2918 -0.0721 0.0217
> CF1_Women..IMP:id18 0.0186 0.0274 0.6814 0.4956 -0.0350 0.0723
> CF1_Women..IMP:id19 -0.0203 0.0637 -0.3189 0.7498 -0.1452 0.1046
>
> ---
> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
>>
>> M0<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP + id, method="ML",
>> data=d) M1<-rma(logOR, logOR.var, mods= ~CF1_Women..IMP * id,
>> method="ML", data=d, btt=c(21:38))
>> anova(M0,M1)
>
> df AIC BIC AICc logLik LRT pval QE tau^2 R^2
> Full 39 515.2691 646.1760 533.4086 -218.6346 488.8555 0.1182
> Reduced 21 520.4230 590.9113 525.2862 -239.2115 41.1539 0.0014 545.9074 0.1510 21.75%
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>
> ---
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
--
Michael
http://www.dewey.myzen.co.uk/home.html
More information about the R-sig-meta-analysis
mailing list