[R-meta] Fail-safe numbers of non-significant effects
Arne Janssen
@rne@j@n@@en @end|ng |rom uv@@n|
Mon Jul 6 16:04:44 CEST 2020
Dear Wolfgang,
The simulation of new effects you suggested seems to work, but in the
case of my data, I will have to add more than 2000 cases to the 68 cases
to get a power of 80%. Also, the observed effect size is slightly
positive (0.085, conf int -0.2798 0.4496), so the simulations assess
the number of cases needed for the effect to become significantly
positive, but the expectation is that the effect should be negative.
I therefore took a slightly different approach: instead of adding new
simulated cases based on the effect size and the s.d. of the entire
dataset, I added simulated cases based on the effect size and s.d. of
all cases with negative effects, resulting in a "fail-safe number" of 35
cases. Obviously, the same could be done with the subset of all cases
with a positive effect. I would highly appreciate your opinion on this.
With best wishes,
Arne Janssen
On 29-Jun-20 16:37, Viechtbauer, Wolfgang (SP) wrote:
> Dear Arne,
>
> Please keep the mailing list in cc.
>
> Indeed, adding 'extreme' effects could drive up the heterogeneity to the point that reaching a significant result becomes difficult or even impossible. And yes, you could fix the variance component(s) to avoid this. Alternatively, instead of adding very large effects, one could add effects that have the same size as the average effect estimated from the initial model. That would have the opposite effect, driving down heterogeneity as more and more such effects are added.
>
> Even better would be an approach where we simulate new effects taking the amount of heterogeneity and the sampling variability into consideration. For a given number of 'new' effects to be added, one would then repeat this many times, checking in what proportion of cases the combined effect is significant. By increasing the number of new effects to be added, one could then figure out how many effects need to be added such that power to find a significant effect is at least 80% (or some other %). Here is an example of this idea:
>
> library(metafor)
>
> yi<- c(0.22, -0.12, 0.41, 0.13, 0.08)
> vi<- c(0.008, 0.002, 0.019, 0.010, 0.0145)
>
> res<- rma(yi, vi, method="DL")
> res
>
> iters<- 1000
>
> maxj<- 20
>
> power<- rep(NA, maxj)
> pvals<- rep(NA, iters)
>
> set.seed(42)
>
> for (j in 1:maxj) {
> print(j)
> for (l in 1:iters) {
> yi.fsn<- c(yi, rnorm(j, coef(res), sqrt(res$tau2 + 1/mean(1/vi))))
> vi.fsn<- c(vi, rep(1/mean(1/vi), j))
> pvals[l]<- rma(yi.fsn, vi.fsn, method="DL")$pval
> }
> power[j]<- mean(pvals<= .05)
> }
>
> plot(1:maxj, power, type="o")
> abline(h=.80, lty="dotted")
> min(which(power>= .80))
>
> So, 15 effects would have to be added to reach 80% power. Note that the line is a bit wiggly, but one could just increase the number of iterations to smooth it out.
>
> Best,
> Wolfgang
>
>
>> -----Original Message-----
>> From: Arne Janssen [mailto:arne.janssen using uva.nl]
>> Sent: Monday, 29 June, 2020 2:25
>> To: Viechtbauer, Wolfgang (SP)
>> Subject: Re: [R-meta] Redundant predictors
>>
>> Dear Wolfgang,
>>
>> I did try something like you suggested just now. With a simple rma it
>> works by simply putting the model calculation in a loop and adding the
>> most extreme case at each step until p< 0.05.
>> When I use rma.mv with a random factor, it does not. The estimate
>> decreases of course, but the s.e. and p increase, so I guess the
>> heterogeneity is increasing much indeed. I am now considering to use the
>> tau of the original model to get a conservative estimate of the numbers
>> of studies needed to get a significant effect. What are your thoughts on
>> this?
>>
>> Thanks a lot in advance.
>>
>> Best wishes,
>> Arne
>>
>> On 28-Jun-20 17:37, Viechtbauer, Wolfgang (SP) wrote:
>>
>>> Dear Arne,
>>>
>>> Your understanding of the fail-safe N is correct, although the way this
>>>
>> number is often computed makes use of Stouffer's method for pooling the p-
>> values of the studies and doesn't actually make use the effect sizes. To
>> illustrate:
>>
>>> library(metafor)
>>>
>>> yi<- c(0.1, 0.07, 0.26, 0.24, 0.19, -0.02, 0.09, -0.04, 0.18, -0.08, -
>>>
>> 0.18, 0.3, -0.09, 0.06, 0.15, -0.05)
>>
>>> vi<- c(0.00943, 0.00134, 0.01923, 0.00962, 0.01449, 0.01613, 0.00585,
>>>
>> 0.0031, 0.01818, 0.0013, 0.01887, 0.01136, 0.00885, 0.00187, 0.00645,
>> 0.01613)
>>
>>> rma(yi, vi, method="FE")
>>>
>>> So a meta-analysis (using a FE model) yields a significant effect. Now
>>>
>> let's compute the fail-safe N:
>>
>>> fsn(yi, vi)
>>>
>>> This says that 35 studies with a null result would yield a non-significant
>>>
>> effect. This approach uses Stouffer's method for pooling the (one-sided) p-
>> values. The p-value for the 16 studies using this method is obtained with:
>>
>>> zi<- yi / sqrt(vi)
>>> pnorm(sum(zi) / sqrt(16), lower.tail=FALSE)
>>>
>>> So, if we would add 35 studies that have (on average) a z-statistic of 0,
>>>
>> then we would get a non-significant pooled p-value:
>>
>>> pnorm(sum(zi) / sqrt(16 + 35), lower.tail=FALSE)
>>>
>>> So that checks out. But note that this makes no reference to effects - it
>>>
>> just uses the p-values and hence z-statistics of the studies.
>>
>>> An approach that is based on the same idea as a FE model is the one by
>>>
>> Rosenberg:
>>
>>> fsn(yi, vi, type="Rosenberg")
>>>
>>> This says that 3 studies with null effects would need to be added to
>>>
>> render the FE model non-significant (where the sampling variances of those 3
>> effects are assumed to be equal to the harmonic mean of the sampling
>> variances of the observed effects). So in other words:
>>
>>> yi.fsn<- c(yi, rep(max(yi), 3))
>>> vi.fsn<- c(vi, rep(1/mean(1/vi), 3))
>>> rma(yi.fsn, vi.fsn, method="FE")
>>>
>>> And indeed, that just fails to be significant at alpha = .05.
>>>
>>> Using the same idea, we could reverse this process. Let's say we start
>>>
>> with these effects, which yield a non-significant result based on a FE model
>> (p = .29):
>>
>>> yi<- c(0.05, 0.07, 0.10, 0.14, 0.02, -0.15, 0.09, -0.04, 0.11, -0.08, -
>>>
>> 0.18, 0.22, -0.09, 0.06, 0.11, -0.05)
>>
>>> rma(yi, vi, method="FE")
>>>
>>> Now by trial-and-error, I can easily figure out that 2 studies with an
>>>
>> effect equal to the maximum observed effect are needed to make the FE model
>> significant:
>>
>>> yi.fsn<- c(yi, rep(max(yi), 2))
>>> vi.fsn<- c(vi, rep(1/mean(1/vi), 2))
>>> rma(yi.fsn, vi.fsn, method="FE")
>>>
>>> If one would want to do this in the context of a RE model, things get more
>>>
>> tricky because one would have to factor in the between-study variance
>> component. So, we start with:
>>
>>> res<- rma(yi, vi)
>>> res
>>>
>>> Now we could assume that the new studies being added come from the same
>>>
>> population of studies and their addition does not alter the estimate of
>> tau^2. Then we again just need to add 2 studies here:
>>
>>> yi.fsn<- c(yi, rep(max(yi), 2))
>>> vi.fsn<- c(vi, rep(1/mean(1/vi), 2))
>>> rma(yi.fsn, vi.fsn, tau2=res$tau2)
>>>
>>> Note that I fix tau^2 to the value obtained from the RE model fitted to
>>>
>> the observed data based on my earlier assumption.
>>
>>> But adding studies with such large effects does actually drive up the
>>>
>> heterogeneity, so if one were to reestimate tau^2, the result would not be
>> significant:
>>
>>> rma(yi.fsn, vi.fsn)
>>>
>>> If we do that, then we would need to add 3 studies:
>>>
>>> yi.fsn<- c(yi, rep(max(yi), 3))
>>> vi.fsn<- c(vi, rep(1/mean(1/vi), 3))
>>> rma(yi.fsn, vi.fsn)
>>>
>>> In case you try something like this with your own data, I would be
>>>
>> interested in hearing what you find.
>>
>>> Best,
>>> Wolfgang
>>>
>>>
>>>> -----Original Message-----
>>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>>>>
>> project.org]
>>
>>>> On Behalf Of Arne Janssen
>>>> Sent: Sunday, 28 June, 2020 16:31
>>>> To: 'r-sig-meta-analysis using r-project.org'
>>>> Subject: [R-meta] Redundant predictors
>>>>
>>>> L.S.,
>>>>
>>>> As far as I understand, the fail-safe N analysis serves to estimate the
>>>> number of cases with zero effect size that would have to be added to
>>>> turn a significant effect size just not significant anymore. Is there
>>>> also an opposite test, i.e. how many cases with significant effect (for
>>>> example the case with the most extreme effect size in the dataset) that
>>>> would have to be added to turn a non-significant effect size into a
>>>> significant one?
>>>>
>>>> Best wishes,
>>>> Arne Janssen
>>>>
> .
>
>
More information about the R-sig-meta-analysis
mailing list