[R-meta] Meta - Bug with REML or small N?

[Viechtbauer, Wolfgang (NP)]

I can't really tell you which one of the estimates is closest to the truth.

On the one hand, one could argue that, given the wide CIs of studies 4 and 5, an estimate of tau^2 close to 0 is potentially reasonable. On the other hand, one could argue that the estimates of studies 4 and 5 are somewhat different from those of the other studies, which might lead to some heterogeneity. One can interpret the plot either way.

Best,
Wolfgang

>-----Original Message-----
>From: Jorge Teixeira [mailto:jorgemmtteixeira using gmail.com]
>Sent: Tuesday, 21 March, 2023 18:32
>To: Viechtbauer, Wolfgang (NP)
>Cc: R Special Interest Group for Meta-Analysis
>Subject: Re: [R-meta] Meta - Bug with REML or small N?
>
>So, did I just win the lottery? 😅
>
>Would you recommend switching to another estimator? DL maybe?
>
>I mean, the t2 value can be "correct", but does not look appropriate when you
>look at the forest plot.
>
>Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>escreveu no dia terça, 21/03/2023 à(s) 17:13:
>So the REML estimate is correct:
>
>dat <- structure(list(study = c("AA", "BB", "CC",
>"EE", "DDD"), en = c(9, 41, 29, 8, 13
>), em = c(32, 27.5, 28.7, 22.8, 30.5), esd = c(1.9, 3.8, 5.2,
>3.8, 4.9), cn = c(8, 26, 28, 10, 14), cm = c(30.1, 24.9, 26.9,
>24.7, 30.2), csd = c(2.4, 3.6, 3, 4.3, 5.1)), row.names = c(NA,
>-5L), class = c("tbl_df", "tbl", "data.frame"))
>
>dat <- escalc(measure="MD", m1i=em, sd1i=esd, n1i=en, m2i=cm, sd2i=csd, n2i=cn,
>data=dat)
>
>res <- rma(dat)
>res
>profile(res, steps=100)
>
>Rather different tau^2 estimates depending on the estimator:
>
>methods <- c("ML","REML","EB","PM","HS","HSk","HE","DL","SJ")
>data.frame(tau2=sapply(methods, \(x) round(rma(dat, method=x)$tau2, 4)))
>
>Sometimes life is difficult.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Jorge Teixeira via R-sig-meta-analysis
>>Sent: Tuesday, 21 March, 2023 17:56
>>To: Michael Dewey
>>Cc: Jorge Teixeira; R Special Interest Group for Meta-Analysis
>>Subject: Re: [R-meta] Meta - Bug with REML or small N?
>>
>>Thanks Michael and Wolfgang.
>>
>>Here goes the df:
>>
>>structure(list(study = c("AA", "BB", "CC",
>>"EE", "DDD"), en = c(9, 41, 29, 8, 13
>>), em = c(32, 27.5, 28.7, 22.8, 30.5), esd = c(1.9, 3.8, 5.2,
>>3.8, 4.9), cn = c(8, 26, 28, 10, 14), cm = c(30.1, 24.9, 26.9,
>>24.7, 30.2), csd = c(2.4, 3.6, 3, 4.3, 5.1)), row.names = c(NA,
>>-5L), class = c("tbl_df", "tbl", "data.frame"))
>>
>>Thanks
>>
>>Michael Dewey <lists using dewey.myzen.co.uk> escreveu no dia terça, 21/03/2023
>>à(s) 16:36:
>>
>>> Dear Jorge
>>>
>>> It looks as though the issue is that REML estimates tau^2 to be
>>> effectively zero whereas DL has a positive estimate. The different ways
>>> of estimating tau^2 do often differ, sometimes by quite surprising
>>> amounts. Whether that is related to the smallish number of studies I
>>> could not say but, at the moment, there does not seem to be evicence of
>>> anything wrong with the implementation of REML.
>>>
>>> Michael
>>>
>>> On 21/03/2023 14:57, Jorge Teixeira via R-sig-meta-analysis wrote:
>>> > Hi.
>>> >
>>> > Let me know if you need me to provide the data for this example.
>>> > Screenshots in the bottom.
>>> >
>>> > I ran this MA with REML, and the weight for random and common effects
>>> > are exactly the same! Never saw anything like this. t2values also don’t
>>> > look plausible.
>>> >
>>> > *_#1_*
>>> > vo2 <- metacont(en  , em, esd, cn, cm, csd, study, method.tau = "REML",
>>> > prediction = TRUE, data = dat_vo2, sm = "MD")
>>> > vo2
>>> >
>>> > Is this a bug or a particular issue of low number of studies and low
>>> > sample size?
>>> >
>>> > *_#2_*
>>> > vo2 <- metacont(en  , em, esd, cn, cm, csd, study, method.tau = "DL",
>>> > prediction = TRUE, data = dat_vo2, sm = "MD")
>>> > vo2
>>> >
>>> > I ran this with DL estimator and weights and t2are plausible. I also ran
>>> > other similar MA using REML and this was all okay.
>>> >
>>> > #1 using SMD instead of MD also looks fine.
>>> >
>>> >
>>> > Thanks,
>>> > Jorge
>>> >
>>> >
>>> > *_REML:_*
>>> >
>>> > image.png
>>> >
>>> >
>>> > *_DL:_*
>>> > image.png