[R-meta] Mismatch between P-value and 95% CI in rma()

[Viechtbauer, Wolfgang (NP)]

It is what it is. The calculation appears correct. For example, for the upper CI bound:

exp(coef(my_metar)[2] + qnorm(.975)*sqrt(vcov(my_metar)[2,2]))

Note that the IRR is for a one-unit increase in the predictor variable. Var1 appears to be some year variable and Var2 some kind of percentage. A one year or one percentage point increase doesn't mean much. You could rescale them with mods = ~ I(Var2/10). Then for Var2 you get:

           estimate ci.lb  ci.ub
I(Var2/10)   0.9395 0.866 1.0193

So, the upper CI corresponds to roughly a 1.02 times (i.e., 2%) increase in the IRR for each 10 percentage points increase in Var2.

Best,
Wolfgang

>-----Original Message-----
>From: Adelina Artenie [mailto:adelina.artenie using bristol.ac.uk]
>Sent: Wednesday, 17 August, 2022 12:08
>To: Viechtbauer, Wolfgang (NP); r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Mismatch between P-value and 95% CI in rma()
>
>Thanks Wolfgang. I got exactly the same as you. I would have expected the P-value
>to be closer to 0.05 or the upper bound of the 95%CI to be bigger than 1.002
>Adelina
>
>From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Date: Wednesday, 17 August 2022 at 05:35
>To: r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
>Cc: Adelina Artenie <adelina.artenie using bristol.ac.uk>
>Subject: RE: [R-meta] Mismatch between P-value and 95% CI in rma()
>Hi Adelina,
>
>Thanks for providing me with the data to check this. For 'Var2', this is the
>model table:
>
>         estimate      se     zval    pval    ci.lb   ci.ub
>intrcpt    2.7086  0.2171  12.4785  <.0001   2.2832  3.1341
>Var2      -0.0062  0.0042  -1.5013  0.1333  -0.0144  0.0019
>
>And this is the exponentiated coefficient for Var2 with the corresponding CI:
>
>     estimate  ci.lb  ci.ub
>Var2   0.9938 0.9857 1.0019
>
>Note that I rounded the values to 4 decimal places (instead of 2), so now we can
>see that the upper bound is above 1 (as it should be). So there is no mismatch.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: Adelina Artenie [mailto:adelina.artenie using bristol.ac.uk]
>>Sent: Tuesday, 16 August, 2022 23:19
>>To: Viechtbauer, Wolfgang (NP)
>>Subject: Re: [R-meta] Mismatch between P-value and 95% CI in rma()
>>
>>ATTACHMENT(S) REMOVED: Ade_dta.xlsx
>>
>>Hi Wolfgang,
>>
>>Thanks for the quick response. I thought I would send you the actual data to
>make
>>sure I get a similar scenario, so just e-mailing you, even if it's anonymised.
>>Please delete once you have looked at it.
>>
>>I attached my code below.
>>Thanks again.
>>Adelina
>>
>>*****
>>*****
>>
>>Ade_dta$Var2<-as.numeric(Ade_dta$Var2, na.strings = "not available")
>>is.numeric(Ade_dta$Var2)
>>
>>my_metar <- rma(measure = "IRLN",
>>                    xi = cases,
>>                    ti = prs_yrs_100,
>>                    add = 0.5,
>>                    to = "only0",
>>                    data = Ade_dta,
>>                    method = "DL",
>>                    test = "z",
>>                    slab = author_inci,
>>                    mods = ~ Var2 ) #Can try Var1 and Var2 here
>>summary(my_metar)
>>round(exp(coef(summary(my_metar))[-1,c("estimate", "ci.lb", "ci.ub")]), 2)
>>
>>*****
>>*****
>>
>>From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>>Date: Tuesday, 16 August 2022 at 21:04
>>To: Adelina Artenie <adelina.artenie using bristol.ac.uk>, r-sig-meta-analysis using r-
>>project.org <r-sig-meta-analysis using r-project.org>
>>Subject: RE: [R-meta] Mismatch between P-value and 95% CI in rma()
>>Hi Adelina,
>>
>>Can you provide a fully reproducible example? Without this, I cannot really tell
>>what is going on.
>>
>>Best,
>>Wolfgang
>>
>>>-----Original Message-----
>>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>>Behalf Of Adelina Artenie
>>>Sent: Tuesday, 16 August, 2022 21:04
>>>To: r-sig-meta-analysis using r-project.org
>>>Subject: [R-meta] Mismatch between P-value and 95% CI in rma()
>>>
>>>Hi everyone,
>>>
>>>I am running a meta-regression analysis for which the outcome is a rate and
>>>several continuous moderator variables. I use rma() in metafor to estimate the
>>>incidence rate ratio (IRR) associated with a 1-unit increase in the moderator.
>>>
>>>If I specify test = "z", my understanding is that both the P-value and the
>95%CI
>>>are estimated using the Wald test (so results should align). However, I see a
>>>small mismatch between the P-value and the 95%CI: the upper bound of the 95% CI
>>>is 1.00 (eg: IRR=0.97; 95%CI: 0.94  1.00), yet the P-value can be anywhere from
>>>P= 0.07 to P= 0.1, depending on the moderator. These P-values seem too high  I
>>>would expect them to be closer to 0.05.
>>>
>>>I wonder if I am making a mistake in my code or interpretation, or
>>alternatively,
>>>if a mismatch like this could occur (though Ive never encountered it before..)
>I
>>>wonder also if this can have anything to do with the log transformation.
>>>
>>>I rely on 95%CI when interpreting my results but I like to also present the P-
>>>value, and so Id like to understand what is happening.
>>>
>>>Many thanks for your help.
>>>Best,
>>>Adelina Artenie
>>>
>>>**
>>>Code:
>>>My_rma <- rma(measure = "IRLN",
>>>                        xi = cases,
>>>                        ti = prs_yrs_100,
>>>                        add = 0.5,
>>>                        to = "only0",
>>>                        data = Incidence_dat,
>>>                        method = "DL",
>>>                        test = "z",
>>>                        slab = author,
>>>                        mods = ~ moderator )
>>>summary(My_rma)
>>>round(exp(coef(summary(My_rma))[-1,c("estimate", "ci.lb", "ci.ub")]), 2)