[R-meta] Appending a risk-of-bias traffic-light plot to a 'three-level' forest plot

[Joshua Bernal]

Dear Dr. Viechtbauer,

Thank you again for your attention and help on the matter.

Best regards,
Josh

On Sat, Feb 19, 2022 at 12:59 AM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> I would suggest to use test="t" with dfs="contain". There is a bit of discussion here:
>
> https://wviechtb.github.io/metafor/reference/rma.mv.html#tests-and-confidence-intervals
>
> Roughly, z-tests tend to be too liberal, which we can counteract by using t-tests. But the default df calculation is quite simplistic and the "contain" method is an improvement on that. Still far from perfect, but a bit better.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: Joshua Bernal [mailto:jdkb9701 using connect.hku.hk]
> >Sent: Friday, 18 February, 2022 17:23
> >To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> >Subject: Re: [R-meta] Appending a risk-of-bias traffic-light plot to a 'three-
> >level' forest plot
> >
> >Dear Dr. Viechtbauer,
> >
> >Thanks very much for your timely help! I'm delighted to learn about a
> >simple alternative approach for creating the traffic light part. I
> >would like to follow-up on the first part of the query on rma.mv /
> >aggregated data...
> >
> >As the help page of the rma.mv function mentions: one can set
> >dfs="contain" (which automatically also sets test="t")
> >
> >I checked to see whether the results would be the same as test="t" if
> >I specify dfs="contain", and whether specifying test="t" with versus
> >without dfs="contain" would yield the same results.
> >
> ># dfs="contain"
> >res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, dfs =
> >"contain", method = "REML", slab = author)
> >
> >estimate      se    zval    pval   ci.lb   ci.ub
> >  0.2495  0.0738  3.3828  0.0007  0.1049  0.3940  ***
> >
> ># test="t"
> >res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, test
> >= "t", method = "REML", slab = author)
> >
> >estimate      se    tval  df    pval   ci.lb   ci.ub
> >  0.2495  0.0738  3.3828  15  0.0041  0.0923  0.4067  **
> >
> ># test="t" and dfs="contain"
> >res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, test
> >= "t", dfs = "contain", method = "REML", slab = author)
> >
> >estimate      se    tval  df    pval   ci.lb   ci.ub
> >  0.2495  0.0738  3.3828   5  0.0196  0.0599  0.4391  *
> >
> >I was wondering why the results differ, and importantly when would it
> >be appropriate or sensible to use each of these approaches?
> >Ultimately, I would like to know how I should determine the
> >appropriate method for this part of the meta-analysis and generally
> >what to consider when doing so (e.g., study sample size, number of
> >studies, number of effect estimates per study)? For example, is it
> >acceptable to use test="z" for a meta-analysis of eight studies with
> >sample sizes of 66, 50, 38, 23, 23, 18, 12, 10 versus five studies
> >with sample sizes of 50, 35, 28, 12, 10; or is it more sensible to be
> >'conservative' and use test="t" in either one (or both) cases?
> >
> >Best regards,
> >Josh