[R-meta] An issue with selmodel( type="step")

[Will Hopkins]

Cool, thanks. I presume enough of those updates were already in the metafor
I downloaded last week (4.5-12), otherwise it wouldn't have run. Is there
another update I should try?

FYI, I tried selmodel(..., type="step", steps=(0.025)) with a similar
simulation of 2500 meta-analyses, all estimates significant, but this time
with small-moderate true heterogeneity (SD=1.5) rather than trivial-small
(SD=0.5). It's still working, although not that well: some remaining bias;
coverage is not very good, in spite of the wide confidence intervals, much
wider than those before adjustment; overall it's better than PEESE. I then
tried type="beta" with the first 100 of the simulations. Again it took more
than 2 hours, and only half of the sims had confidence limits for the fixed
effects. Overall beta would be better, if there was some way to coax it to
give more confidence limits.

Others have already noted how heterogeneity reduces the effectiveness of
adjustment for publication bias, so it's obviously important to include
study and subject characteristics as predictors to try to reduce
heterogeneity to a trivial value.

Will

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On
Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
Sent: Thursday, March 28, 2024 1:10 AM
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis using r-project.org>
Cc: Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer using maastrichtuniversity.nl>
Subject: Re: [R-meta] An issue with selmodel( type="step")

I am working on some updates to selmodel() and one of those changes is that
the function now continues to run even if an interval contains no p-values.
As discussed previously, the corresponding delta estimate will then either
try to drift to 0 or to infinity. Results should be treated with caution (as
noted in the output from the function).

Best,
Wolfgang

> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> 
> On Behalf Of Will Hopkins via R-sig-meta-analysis
> Sent: Tuesday, March 26, 2024 20:47
> To: 'R Special Interest Group for Meta-Analysis' 
> <r-sig-meta-analysis using r- project.org>
> Cc: Will Hopkins <willthekiwi using gmail.com>
> Subject: [R-meta] An issue with selmodel( type="step")
>
> Wolfgang, according to the documentation "there must be at least one 
> observed p-value within each interval to fit this model. If this is 
> not the case, an error will be issued." When I tried it with 
> steps=(0.025) for a simulated meta-analysis in which all estimates 
> were significant (p<0.05), it issued an error ("One or more intervals 
> do not contain any observed p-values"), but the analysis nevertheless 
> produced a result. When I ran it with 2500 such simulations, it 
> produced point estimates for the fixed effects with 2456 simulations, 
> and confidence limits with 2399. See below for a typical result. The 
> selection model results show 0.0000 for the estimated probability of 
> non-significant p values, as expected, so how is it able to make
adjustments?
>
> Type="steps" seems to be about as good as type="beta" for bias and 
> coverage with this particular set of study characteristics, but the 
> beta type runs so slowly that I have only used it for 100 simulations for
comparison so far.
> It took more than an hour, and only 51/100 produced confidence limits 
> for the fixed effects, so in these respects the steps type is 
> definitely better).
>
> Will
>
> Mixed-Effects Model (k = 22; tau^2 estimator: ML)
>
> tau^2 (estimated amount of residual heterogeneity): 0.2541 (SE = 0.2288)
> tau (square root of estimated tau^2 value):         0.5041
>
> Test for Residual Heterogeneity:
> LRT(df = 1) = 4.9738, p-val = 0.0257
>
> Test of Moderators (coefficients 1:2):
> QM(df = 2) = 94.2813, p-val < .0001
>
> Model Results:
>
>                estimate      se    zval    pval    ci.lb   ci.ub
> xxx$SexFemale    2.9331  0.3022  9.7046  <.0001   2.4359  3.4302  ***
> xxx$SexMale      1.1294  0.7376  1.5312  0.1257  -0.0838  2.3425
>
> Test for Selection Model Parameters:
> LRT(df = 1) = 22.8488, p-val < .0001
>
> Selection Model Results:
>
>                      k  estimate   se  zval  pval  ci.lb  ci.ub
> 0     < p <= 0.025  22    1.0000  ---   ---   ---    ---    ---
> 0.025 < p <= 1       0    0.0000   NA    NA    NA     NA     NA

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org To
manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis