[R-meta] Error: Ratio of largest to smallest sampling variance extremely large

[James Pustejovsky]

I agree with Wolfgang's suggestion that it would be important to conduct
sensitivity analysis here. Specifically, I would want to see what happens
if the smallest sampling variances are set to some lower bound. For
example, if there is just one effect size with a very small sampling
variance, try replacing the actual sampling variance with the next largest
sampling variance (or if there are several very small, outlying variances,
replace them all with the minimum of the remaining sampling variances.
Hopefully this won't have much consequence on the overall average effect
size. But I would anticipate that the between-study heterogeneity estimate
could be quite sensitive.

James

On Fri, Feb 15, 2019 at 10:08 AM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Hi Florencia,
>
> As the warning says, the results may not be stable. To be precise, the
> function checks if:
>
> max(vi) / min(vi) >= 1e7
>
> So, that would be the case for example if the largest sampling variance is
> 1 and the smallest is .0000001 (or even smaller). For many effect size
> measures, such a discrepancy would usually be pretty indicative of a data
> entry/coding error, so this is why James was suggesting to check the data.
>
> You mentioned using log response ratios, where the usual equation for the
> sampling variance is:
>
> vi <- sd1i^2/(n1i*m1i^2) + sd2i^2/(n2i*m2i^2)
>
> Another way to write this is:
>
> vi <- cv1i^2 / n1i + cv2i^2 / n2i
>
> where cv1i and cv2i are the coefficient of variation values for the two
> groups. So, either the sample sizes vary by many many magnitudes over
> studies in your data (e.g., one study with 10 observations, another with
> 100000000) or the coefficient of variation values differ greatly (or a
> combination of the two). The former seems unlikely to me, the latter would
> make me question whether the corresponding lrr values should even be
> combined.
>
> This aside, I cannot tell you whether the warning can be ignored or not in
> your particular case. The reason why I put in the check is that I saw some
> numerical issues with the internal algorithms in rma.mv() that were
> sometimes triggered by having a V matrix that is very ill-conditioned (
> https://en.wikipedia.org/wiki/Condition_number). Using max(vi) / min(vi)
> >= 1e7 is a very simplified check for this.
>
> If you have just a few cases where vi is very very small (or large), you
> could see what happens with the results if you shrink those vi values a bit
> up (or down). Hopefully results do not change drastically. That would be a
> good sign that things are alright.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: florencia miguel [mailto:mflormiguel using gmail.com]
> Sent: Friday, 15 February, 2019 16:11
> To: Viechtbauer, Wolfgang (SP)
> Cc: James Pustejovsky; R meta
> Subject: Re: [R-meta] Error: Ratio of largest to smallest sampling
> variance extremely large
>
> Thanks James and Wolfgang.
>
> James, I´ve checked data and I have some very small sampling variances.
> BUT, these are not errors, it is the structure of data we are working with.
>
> Wolfgang, I installed that version of metafor, i did get warnings when
> running models, is that ok?
>
> Best
> Florencia
>
> El jue., 14 de feb. de 2019 a la(s) 19:29, Viechtbauer, Wolfgang (SP) (
> wolfgang.viechtbauer using maastrichtuniversity.nl) escribió:
> Hi Florencia,
>
> If you install the 'devel' version of metafor (
> https://wviechtb.github.io/metafor/#installation), then you should get a
> warning but no longer an error. However, the warning is there for a reason;
> the results might not be trustworthy.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On Behalf Of James Pustejovsky
> Sent: Thursday, 14 February, 2019 22:46
> To: florencia miguel
> Cc: R meta
> Subject: Re: [R-meta] Error: Ratio of largest to smallest sampling
> variance extremely large
>
> Florencia,
>
> I think the issue might have more to do with your data than with the
> estimation procedures. Could you try running one or all of the following
> lines:
>
> summary(mdata.all$var.es)
> plot(density(mdata.all$var.es)
> bottom_n(mdata.all, 5, var.es)
>
> This will provide a five-number summary of the sampling variances of your
> effect size estimates. If you have some that are very very small, this will
> cause the error you seem to have encountered. It might be worth checking
> the summary statistics for the effect sizes with very small variances, to
> see if there are data entry errors, or reporting errors in the primary
> sources.
>
> James
>
> On Thu, Feb 14, 2019 at 1:57 PM florencia miguel <mflormiguel using gmail.com>
> wrote:
>
> > Dear all, I am running a meta analysis with the main aim of comparing
> three
> > different kind of interventions and four kind of outcomes. I want to
> > perform different models for interventions and outcomes. I could run
> random
> > effects models using the package meta but, as I need to include
> moderators
> > in the models I tryed the metafor package.
> >
> > The problem is that I obtained this error when running rma.uni and
> > rma.mv functions:
> > "Error in rma.mv(yi = lrr, V = var.es, mods = ~aridity.index, method =
> > "REML", : Ratio of largest to smallest sampling variance extremely large.
> > Cannot obtain stable results."
> >
> > I am using Log response ratio as effect sizes. I know that data are very
> > heterogeneous (some rows with high variances values and other with low
> > variances) because I am comparing different kind of measures. So, I
> > performed models by subgroups (subsetting by interventions), and I
> obtained
> > the same type of error.
> >
> > Here are some codes:
> >
> > mod1 <- rma(lrr, var.es, mods= aridity.index, data=mdata.all,
> > subset=intervention=="vegetation")
> >
> > mod.2<-rma.mv(yi=lrr, V=var.es, mods= aridity.index, method = "REML",
> > test="t", random = ~ 1 | ID, data=mdata.all, sparse=TRUE)
> >
> > mod.3 <- rma(lrr, var.es, mods=  ~intervention, data = mdata.all,
> subset =
> > paradigm == "active")
> >
> > ##filtering by interventions
> > mdata.veg <- mydata %>%
> >   filter(intervention=="vegetation") %>%
> >   filter(!is.na(lrr)) %>%
> >   filter(!is.na(var.es))
> >
> > mod<-rma(lrr,var.es, mods= aridity.index, digits=4,data=mdata.veg)
> >
> > I dont know why i getting the same error after subsetting or filtering by
> > groups.
> >
> > Thank you in advance!
> > Florencia
>

	[[alternative HTML version deleted]]