[R-meta] Unrealistic confidence limits for heterogeneity?

[Viechtbauer, Wolfgang (NP)]

Interesting point.

Just to note: Right now, a warning is already issued when tau2.min is set to a value lower than -min(vi) (as it then automatically sets tau2.min to -min(vi)). This is also documented (https://wviechtb.github.io/metafor/reference/rma.uni.html#note-1):

rma(yi, vi, data=dat, control=list(tau2.min=-1))
[...]
Warning message:
Value of 'tau2.min' constrained to -min(vi) = -0.1212.

Also, the function prevents shenanigans like trying to force tau^2 to be smaller than -min(vi):

rma(yi, vi, data=dat, tau2=-1)

Error in rma(yi, vi, data = dat, tau2 = -1) :
  Some marginal variances are negative.

But if the *estimate* is equal to tau2.min, then there is no further note about this (except that the user will see that in the output). In a way, this is no different than when tau^2 is estimated to be 0 (under the default tau2.min=0 setting).

Technically, it is not even possible to fit the model with tau2 = -min(vi) with ML/REML estimation, since this will lead to division by zero:

rma(yi, vi, data=dat, tau2=-min(vi))

Error in rma(yi, vi, data = dat, tau2 = -min(vi)) :
  Division by zero when computing the inverse variance weights.

The step halving algorithm that is used as part of ML/REML estimation in rma.uni() should essentially prevent that the estimate can actually hit -min(vi).

One can of course construct a dataset where the likelihood profile peaks essentially at -min(vi) (or rather, just a smidge above it), which might suggest that a value of tau^2 that is even smaller than -min(vi) could lead to an even larger likelihood, but then one would have to allow negative weights.

Best,
Wolfgang

>-----Original Message-----
>From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]
>Sent: Sunday, 02 April, 2023 12:10
>To: R Special Interest Group for Meta-Analysis
>Cc: Viechtbauer, Wolfgang (NP)
>Subject: Re: [R-meta] Unrealistic confidence limits for heterogeneity?
>
>This raises the question of what to do when an estimate is at the
>boundary of the parameter space (or beyond it). Is it worth warning the
>user that this may have happened (ie the lower bound is at -min(vi)? It
>suggests to me that the user has here an attempt to fit a model to a
>dataset which is inadequate for that model.
>
>Michael
>
>On 01/04/2023 13:17, Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
>wrote:
>> Hi James,
>>
>> Your example made me realize some tricky issues when tau2.min is set to -
>min(vi) (which is the smallest value of tau2.min allowed, since otherwise vi +
>tau^2 could become negative) and one then wants to compute CIs for tau^2 (in your
>example, confint() fails with res2). In the devel version, I now push tau2.min
>just slightly above -min(vi) in confint() that hopefully deals with this issue (a
>bit of a hack but seems to work, at least in your example).
>>
>> res2 <- rma(yi, vi, data=dat, control=list(tau2.min=-min(vi)))
>> res2
>> confint(res2)
>> confint(res2, type="PL")
>> res2 <- rma(yi, vi, data=dat, method="GENQ", weights=1/vi,
>control=list(tau2.min=-min(vi)))
>> confint(res2)
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org]
>On
>>> Behalf Of James Pustejovsky via R-sig-meta-analysis
>>> Sent: Thursday, 30 March, 2023 19:00
>>> To: R Special Interest Group for Meta-Analysis
>>> Cc: James Pustejovsky
>>> Subject: Re: [R-meta] Unrealistic confidence limits for heterogeneity?
>>>
>>> Some further comments in addition to Michael's response below.
>>>
>>> James
>>>
>>> 1. It is possible to allow for negative heterogeneity estimates using the
>>> metafor package. Here is an example of the syntax:
>>>
>>> library(metafor)
>>>
>>> # generate data with no heterogeneity
>>> set.seed(20230330)
>>> k <- 10
>>> vi <- 4 / (rpois(k, 22) + 8)
>>> yi <- rnorm(k, mean = 0.2, sd = sqrt(vi))
>>> dat <- data.frame(yi, vi)
>>>
>>> # regular random effects meta-analysis, REML estimator
>>> res1 <- rma(yi = yi, vi = vi, data=dat)
>>> res1
>>>
>>> # allow negative heterogeneity, REML estimator
>>> res2 <- rma(yi = yi, vi = vi, data=dat, control=list(tau2.min=-min(vi)))
>>> res2
>>>
>>> # allow negative heterogeneity, other heterogeneity estimators
>>> rma(yi = yi, vi = vi, data=dat, method = "ML",
>>> control=list(tau2.min=-min(vi)))
>>> rma(yi = yi, vi = vi, data=dat, method = "DL",
>>> control=list(tau2.min=-min(vi)))
>>> rma(yi = yi, vi = vi, data=dat, method = "HE",
>>> control=list(tau2.min=-min(vi)))
>>>
>>> 2. You can obtain the estimated standard error for tau-squared as follows:
>>> res1$se.tau2
>>> res2$se.tau2
>>>
>>> 3. The metafor package implements several different confidence intervals
>>> for tau-squared. The GENQ method requires estimating the model with method
>>> GENQ.
>>> confint(res1) # confidence interval for tau-squared
>>> confint(res1, type = "PL") # profile likelihood method
>>> confint(res1, type = "QP") # Q-profile method
>>> rma(yi = yi, vi = vi, data=dat, weights = 1 / vi, method = "GENQ")  |>
>>>   confint(type = "GENQ") # Generalized Q-statistic method