[R-meta] Calculation of p values in selmodel

[Viechtbauer, Wolfgang (NP)]

Please always respond to the list, not just the individual that replied to you.

Halving the p-value from two-sided tests is not the right way to compute one-sided p-values.

Say you do an independent samples t-test with H1: mu1 > mu2 versus H0: mu1 <= mu2. Then:

pt(2.34, df=20, lower.tail=FALSE)

and

pt(-2.34, df=20, lower.tail=FALSE)

will give you the correct one-sided p-values, depending on whether mean1 > mean2 (in the first case) or mean1 < mean2 (in the second case).

In a two-sided test (i.e., H1: mu1 != mu2 versus H0: mu1 = mu2), we would compute the p-value with:

2*pt(abs(2.34), df=20, lower.tail=FALSE)
2*pt(abs(-2.34), df=20, lower.tail=FALSE)

for these two cases, but dividing these by 2 does not work in the second case.

If you use selmodel(..., alternative="greater"), then you really should also pass one-sided p-values to the function, where the p-values are computed for an alternative hypothesis with the appropriate directionality.

Best,
Wolfgang

> -----Original Message-----
> From: Will Hopkins <willthekiwi using gmail.com>
> Sent: Friday, March 29, 2024 01:04
> To: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Cc: 'Will Hopkins' <willthekiwi using gmail.com>
> Subject: RE: [R-meta] Calculation of p values in selmodel
>
> Oh, I just assumed that it was appropriate to pass the usual p value into
> selmodel with your new pval= option. Halving the p value did the trick.
>
> I ran it with 2172 simulations in which 90% of non-significant effects were
> omitted. The coverage and confidence limits were not quite as good, but
> practically the same, as with the usual method. The usual method produced
> confidence limits in 2018 of the 2172 sims, whereas the pval method produced
> them in 2007, a negligible difference.  I had downloaded the latest metafor
> from github, and it's showing 4.7-0.
>
> Thanks again for your expertise and engagement, Wolfgang!
>
> Will
>
> -----Original Message-----
> From: Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Sent: Friday, March 29, 2024 12:41 AM
> To: R Special Interest Group for Meta-Analysis
> <r-sig-meta-analysis using r-project.org>
> Cc: Will Hopkins <willthekiwi using gmail.com>
> Subject: RE: [R-meta] Calculation of p values in selmodel
>
> If you passed two-sided p-values to the function but the simulated selection
> process was based on the significance of one-sided tests (i.e., the
> significance plus the direction of the effects), then this doesn't match up
> and it should not be a surprise then that the model cannot correct for the
> selection process.
>
> Best,
> Wolfgang