[R-meta] Asking for continuous moderating effects

[Viechtbauer, Wolfgang (SP)]

Thanks. The problem stems from the random-effects model (which needs to be fit to compute R^2) not converging. We can see this if we actually fit the model:

res <- rma(rAB, vAB, data=sample_moderator)

We can get it to converge by increasing the number of iterations:

res <- rma(rAB, vAB, data=sample_moderator, control=list(maxit=1000))

And then we do the same in the meta-regression model:

res <- rma(rAB, vAB, mods = ~ M1, data=sample_moderator, control=list(maxit=1000))
res

And now you got the R^2 value.

This wasn't your question, and I don't know any of the details of this analysis, but since I can see the dataset now, I would have several concerns about this analysis.

First, it seems like there are really two studies, the first providing 16 correlations, the second providing one. I do not know if the N=34 for the first study always refers to the same 34 individuals, but if so, these correlations definitely cannot be treated as independent (which they are at this point). Even if there is no overlap of participants, it's a bit of an unusual data structure.

Second, the 16 correlations from the first study are really four times the same set of four correlations. That is also rather peculiar.

Finally, a bubble plot based on the meta-regression model (regplot(res)) shows that the entire relationship between 'M1' and 'rAB' hinges on that single correlation from the second study. plot(influence(res)) shows how influential that one data point is.

Again, I do not know any of the details, but these data look very strange, the conclusion (that there is a relationship between M1 and rAB) is not robust, and the analysis is probably ignoring important dependencies.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Kim, Jaewoo
>Sent: Friday, 10 September, 2021 0:18
>To: r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Asking for continuous moderating effects
>
>ATTACHMENT(S) REMOVED: sample_moderator.R | Moderator - sample data.xlsx
>
>Thank you for your heads up, Professor Viechtbauer. I will keep that in mind. I
>have attached the sample excel file and the sample code where the problem is to be
>reproduced. Could you please take a look at these files? As I have a problem with
>the result I below, I only included the sample data regarding result I. Please let
>me know if you need anything else to reproduce. Many thanks.
>
>I am analyzing continuous moderating effects with metafor. When I ran the
>moderator analysis, I found that R^2 is NA%. Since my metafor is an older version,
>I thought NA appeared in the results. Though I updated my metafor (v 3.0-2), the
>results were still the same (please see the result I). However, when I ran another
>moderator analysis with metafor, R^2 appeared 0, not NA (please see result II).
>May I ask why this happens? In this case, can I view NA as 0%? Below are the
>results with R code.
>
>Results I
>
># Moderator 1
>res <- rma(rAB, vAB, mods = ~ M1, data=sample_moderator)
>res
>
>Mixed-Effects Model (k = 17; tau^2 estimator: REML)
>
>tau^2 (estimated amount of residual heterogeneity):     0.0069 (SE = 0.0112)
>tau (square root of estimated tau^2 value):             0.0830
>I^2 (residual heterogeneity / unaccounted variability): 22.42%
>H^2 (unaccounted variability / sampling variability):   1.29
>R^2 (amount of heterogeneity accounted for):            NA%
>
>Test for Residual Heterogeneity:
>QE(df = 15) = 16.9198, p-val = 0.3237
>
>Test of Moderators (coefficient 2):
>QM(df = 1) = 9.4482, p-val = 0.0021
>
>Model Results:
>
>         estimate      se     zval    pval    ci.lb    ci.ub
>intrcpt   -0.5843  0.2530  -2.3094  0.0209  -1.0802  -0.0884   *
>M1         0.2672  0.0869   3.0738  0.0021   0.0968   0.4375  **
>
>===================================================================
>
>Results II
>
>res <- rma(rAE, vAE, mods = ~ M1, data=sample_moderator)
>res
>
>Mixed-Effects Model (k = 10; tau^2 estimator: REML)
>
>tau^2 (estimated amount of residual heterogeneity):     0.0291 (SE = 0.0194)
>tau (square root of estimated tau^2 value):             0.1705
>I^2 (residual heterogeneity / unaccounted variability): 77.99%
>H^2 (unaccounted variability / sampling variability):   4.54
>R^2 (amount of heterogeneity accounted for):            0.00%
>
>Test for Residual Heterogeneity:
>QE(df = 8) = 37.9082, p-val < .0001
>
>Test of Moderators (coefficient 2):
>QM(df = 1) = 0.9967, p-val = 0.3181
>
>Model Results:
>
>         estimate      se     zval    pval    ci.lb   ci.ub
>intrcpt    0.4235  0.2087   2.0297  0.0424   0.0146  0.8325  *
>M1        -0.0862  0.0863  -0.9984  0.3181  -0.2554  0.0830
>
>========================================================================
>
>From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: Thursday, September 9, 2021 3:05 AM
>To: Kim, Jaewoo <jkim using bauer.uh.edu>; r-sig-meta-analysis using r-project.org <r-sig-
>meta-analysis using r-project.org>
>Subject: RE: [R-meta] Asking for continuous moderating effects
>
>Dear Jaewoo,
>
>Please provide a reproducible example for the case where R^2 is reported as NA%.
>
>Also, please switch of line wrapping in your email client (or whatever is causing
>the results below to be so mangled up).
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Kim, Jaewoo
>>Sent: Thursday, 09 September, 2021 9:46
>>To: r-sig-meta-analysis using r-project.org
>>Subject: [R-meta] Asking for continuous moderating effects
>>
>>Hello,
>>
>>Hope that this email goes well. I am analyzing continuous moderating effects with
>>metafor. When I ran the moderator analysis, I found that R^2 is NA%. Since my
>>metafor is an older version, I thought NA appeared in the results. Though I
>>updated my metafor (v 3.0-2), the results were the same (please see the result
>I).
>>However, when I ran another moderator analysis with metafor, R^2 appeared 0, not
>>NA (please see result II). May I ask why this happens? In this case, can I view
>NA
>>as 0%? Below are the results with R code.
>>
>>Results I
>>
>>res <-
>>rma(rAB, vAB, mods = ~ M1, data=en_moderator)
>>res
>>
>>Mixed-Effects
>>Model (k = 17; tau^2 estimator: REML)
>>
>>tau^2
>>(estimated amount of residual heterogeneity):     0.0069 (SE = 0.0112)
>>tau (square
>>root of estimated tau^2 value):
>>0.0830
>>I^2 (residual
>>heterogeneity / unaccounted variability): 22.42%
>>H^2
>>(unaccounted variability / sampling variability):   1.29
>>R^2 (amount
>>of heterogeneity accounted for):
>>NA%
>>
>>Test for
>>Residual Heterogeneity:
>>QE(df = 15) =
>>16.9198, p-val = 0.3237
>>
>>Test of
>>Moderators (coefficient 2):
>>QM(df = 1) =
>>9.4482, p-val = 0.0021
>>
>>Model
>>Results:
>>
>>         estimate      se
>>zval    pval    ci.lb
>>ci.ub
>>intrcpt   -0.5843
>>0.2530  -2.3094  0.0209
>>-1.0802  -0.0884   *
>>M1         0.2672
>>0.0869   3.0738  0.0021
>>0.0968   0.4375  **
>>
>>===================================================================
>>
>>Results II
>>
>>res <-
>>rma(rAE, vAE, mods = ~ M1, data=en_moderator)
>>res
>>
>>Mixed-Effects
>>Model (k = 10; tau^2 estimator: REML)
>>
>>tau^2
>>(estimated amount of residual heterogeneity):     0.0291 (SE = 0.0194)
>>tau (square
>>root of estimated tau^2 value):
>>0.1705
>>I^2 (residual
>>heterogeneity / unaccounted variability): 77.99%
>>H^2
>>(unaccounted variability / sampling variability):   4.54
>>R^2 (amount
>>of heterogeneity accounted for):
>>0.00%
>>
>>Test for
>>Residual Heterogeneity:
>>QE(df = 8) =
>>37.9082, p-val < .0001
>>
>>Test of
>>Moderators (coefficient 2):
>>QM(df = 1) =
>>0.9967, p-val = 0.3181
>>
>>Model
>>Results:
>>
>>         estimate      se
>>zval    pval    ci.lb
>>ci.ub
>>intrcpt    0.4235
>>0.2087   2.0297  0.0424
>>0.0146  0.8325  *
>>M1        -0.0862
>>0.0863  -0.9984  0.3181
>>-0.2554  0.0830