[R-meta] An issue with selmodel( type="step")

[Viechtbauer, Wolfgang (NP)]

There was another update (4.5-13) where I also added a new argument to selmodel() called 'decreasing' -- when set to TRUE, it forces the delta values to be a monotonically decreasing function of the p-values. And I just pushed another update, but this one was mostly just to update metafor to version 4.6-0 for a new CRAN release.

If you simulated only significant estimates, then one should heed the warning that is issued by the function that results from step function models with empty intervals should be treated with caution.

As for the beta selection model -- it isn't easy to make this model fit (as you noticed) due to some numerical issues when working with a beta distribution. When there are p-values very close to 0, then this can lead to selection weights that want to drift off to infinity. There is a note about this in the docs:

https://wviechtb.github.io/metafor/reference/selmodel.html#note-1

So one has to use numerical fixes to avoid this, which isn't ideal.

As for heterogeneity -- yes, it is difficult to distinguish selection effects and heterogeneity. Random-effects selection models are notoriously difficult to fit and typically require a very large number of studies to yield estimates that are somewhat usable/stable.

Best,
Wolfgang

> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Will Hopkins via R-sig-meta-analysis
> Sent: Thursday, March 28, 2024 08:56
> To: 'R Special Interest Group for Meta-Analysis' <r-sig-meta-analysis using r-
> project.org>
> Cc: Will Hopkins <willthekiwi using gmail.com>
> Subject: Re: [R-meta] An issue with selmodel( type="step")
>
> Cool, thanks. I presume enough of those updates were already in the metafor
> I downloaded last week (4.5-12), otherwise it wouldn't have run. Is there
> another update I should try?
>
> FYI, I tried selmodel(..., type="step", steps=(0.025)) with a similar
> simulation of 2500 meta-analyses, all estimates significant, but this time
> with small-moderate true heterogeneity (SD=1.5) rather than trivial-small
> (SD=0.5). It's still working, although not that well: some remaining bias;
> coverage is not very good, in spite of the wide confidence intervals, much
> wider than those before adjustment; overall it's better than PEESE. I then
> tried type="beta" with the first 100 of the simulations. Again it took more
> than 2 hours, and only half of the sims had confidence limits for the fixed
> effects. Overall beta would be better, if there was some way to coax it to
> give more confidence limits.
>
> Others have already noted how heterogeneity reduces the effectiveness of
> adjustment for publication bias, so it's obviously important to include
> study and subject characteristics as predictors to try to reduce
> heterogeneity to a trivial value.
>
> Will
>
> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On
> Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
> Sent: Thursday, March 28, 2024 1:10 AM
> To: R Special Interest Group for Meta-Analysis
> <r-sig-meta-analysis using r-project.org>
> Cc: Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: Re: [R-meta] An issue with selmodel( type="step")
>
> I am working on some updates to selmodel() and one of those changes is that
> the function now continues to run even if an interval contains no p-values.
> As discussed previously, the corresponding delta estimate will then either
> try to drift to 0 or to infinity. Results should be treated with caution (as
> noted in the output from the function).
>
> Best,
> Wolfgang
>
> > -----Original Message-----
> > From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org>
> > On Behalf Of Will Hopkins via R-sig-meta-analysis
> > Sent: Tuesday, March 26, 2024 20:47
> > To: 'R Special Interest Group for Meta-Analysis'
> > <r-sig-meta-analysis using r- project.org>
> > Cc: Will Hopkins <willthekiwi using gmail.com>
> > Subject: [R-meta] An issue with selmodel( type="step")
> >
> > Wolfgang, according to the documentation "there must be at least one
> > observed p-value within each interval to fit this model. If this is
> > not the case, an error will be issued." When I tried it with
> > steps=(0.025) for a simulated meta-analysis in which all estimates
> > were significant (p<0.05), it issued an error ("One or more intervals
> > do not contain any observed p-values"), but the analysis nevertheless
> > produced a result. When I ran it with 2500 such simulations, it
> > produced point estimates for the fixed effects with 2456 simulations,
> > and confidence limits with 2399. See below for a typical result. The
> > selection model results show 0.0000 for the estimated probability of
> > non-significant p values, as expected, so how is it able to make
> adjustments?
> >
> > Type="steps" seems to be about as good as type="beta" for bias and
> > coverage with this particular set of study characteristics, but the
> > beta type runs so slowly that I have only used it for 100 simulations for
> comparison so far.
> > It took more than an hour, and only 51/100 produced confidence limits
> > for the fixed effects, so in these respects the steps type is
> > definitely better).
> >
> > Will
> >
> > Mixed-Effects Model (k = 22; tau^2 estimator: ML)
> >
> > tau^2 (estimated amount of residual heterogeneity): 0.2541 (SE = 0.2288)
> > tau (square root of estimated tau^2 value):         0.5041
> >
> > Test for Residual Heterogeneity:
> > LRT(df = 1) = 4.9738, p-val = 0.0257
> >
> > Test of Moderators (coefficients 1:2):
> > QM(df = 2) = 94.2813, p-val < .0001
> >
> > Model Results:
> >
> >                estimate      se    zval    pval    ci.lb   ci.ub
> > xxx$SexFemale    2.9331  0.3022  9.7046  <.0001   2.4359  3.4302  ***
> > xxx$SexMale      1.1294  0.7376  1.5312  0.1257  -0.0838  2.3425
> >
> > Test for Selection Model Parameters:
> > LRT(df = 1) = 22.8488, p-val < .0001
> >
> > Selection Model Results:
> >
> >                      k  estimate   se  zval  pval  ci.lb  ci.ub
> > 0     < p <= 0.025  22    1.0000  ---   ---   ---    ---    ---
> > 0.025 < p <= 1       0    0.0000   NA    NA    NA     NA     NA