[R-meta] Appending a risk-of-bias traffic-light plot to a 'three-level' forest plot

[Viechtbauer, Wolfgang (SP)]

I would suggest to use test="t" with dfs="contain". There is a bit of discussion here:

https://wviechtb.github.io/metafor/reference/rma.mv.html#tests-and-confidence-intervals

Roughly, z-tests tend to be too liberal, which we can counteract by using t-tests. But the default df calculation is quite simplistic and the "contain" method is an improvement on that. Still far from perfect, but a bit better.

Best,
Wolfgang

>-----Original Message-----
>From: Joshua Bernal [mailto:jdkb9701 using connect.hku.hk]
>Sent: Friday, 18 February, 2022 17:23
>To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Appending a risk-of-bias traffic-light plot to a 'three-
>level' forest plot
>
>Dear Dr. Viechtbauer,
>
>Thanks very much for your timely help! I'm delighted to learn about a
>simple alternative approach for creating the traffic light part. I
>would like to follow-up on the first part of the query on rma.mv /
>aggregated data...
>
>As the help page of the rma.mv function mentions: one can set
>dfs="contain" (which automatically also sets test="t")
>
>I checked to see whether the results would be the same as test="t" if
>I specify dfs="contain", and whether specifying test="t" with versus
>without dfs="contain" would yield the same results.
>
># dfs="contain"
>res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, dfs =
>"contain", method = "REML", slab = author)
>
>estimate      se    zval    pval   ci.lb   ci.ub
>  0.2495  0.0738  3.3828  0.0007  0.1049  0.3940  ***
>
># test="t"
>res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, test
>= "t", method = "REML", slab = author)
>
>estimate      se    tval  df    pval   ci.lb   ci.ub
>  0.2495  0.0738  3.3828  15  0.0041  0.0923  0.4067  **
>
># test="t" and dfs="contain"
>res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, test
>= "t", dfs = "contain", method = "REML", slab = author)
>
>estimate      se    tval  df    pval   ci.lb   ci.ub
>  0.2495  0.0738  3.3828   5  0.0196  0.0599  0.4391  *
>
>I was wondering why the results differ, and importantly when would it
>be appropriate or sensible to use each of these approaches?
>Ultimately, I would like to know how I should determine the
>appropriate method for this part of the meta-analysis and generally
>what to consider when doing so (e.g., study sample size, number of
>studies, number of effect estimates per study)? For example, is it
>acceptable to use test="z" for a meta-analysis of eight studies with
>sample sizes of 66, 50, 38, 23, 23, 18, 12, 10 versus five studies
>with sample sizes of 50, 35, 28, 12, 10; or is it more sensible to be
>'conservative' and use test="t" in either one (or both) cases?
>
>Best regards,
>Josh