[R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11

[James Pustejovsky]

Hi Tarun,

Here is a tutorial paper on RVE in R:

Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016). Handling complex
meta-analytic data structures using robust variance estimates: A tutorial
in R. *Journal of Developmental and Life-Course Criminology*, *2*(1),
85-112.

The small-sample adjustments that Wolfgang mentioned are implemented in the
robumeta and clubSandwich packages. The methods implemented in these
packages are described here:

Tipton, E., & Pustejovsky, J. E. (2015). Small-sample adjustments for tests
of moderators and model fit using robust variance estimation in
meta-regression. *Journal of Educational and Behavioral Statistics*, *40*(6),
604-634. https://www.jepusto.com/publication/rve-for-meta-regression/

The clubSandwich method also has a vignette demonstrating its syntax for
use with meta-analysis/meta-regression models:

https://cran.r-project.org/web/packages/clubSandwich/vignettes/meta-analysis-with-CRVE.html


Kind Regards,
James

On Thu, Apr 23, 2020 at 6:38 AM Tarun Khanna <khanna using hertie-school.org>
wrote:

> Thank you Wolfgang. Would you recommend any resources that explain in more
> detail the implementation of RVE in R using the resources that you
> mentioned?
>
> Warm regards,
> Tarun Khanna
>
> > On 23. Apr 2020, at 13:01, Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >
> > With respect to 1: I am not aware of any theoretical work or simulation
> studies that show that DL has better statistical properties than REML in
> small samples. What studies does Ringquist (2013) cite to support this
> claim?
> >
> > With respect to 2: Indeed, RVE is a method that works in large samples.
> The issue with treating the ratio of an estimate to its standard error as a
> z-value applies to pretty much all meta-analytic methods, whether you use
> RVE or not. Improved methods (such as the K&H adjustment) or small-sample
> corrections for RVE have been developed that can mitigate this problem.
> Typically, these methods then lead to using the t-distribution (with some
> estimated degrees of freedom) as the test statistic.
> >
> > And no, just using rma.mv() doesn't do RVE. You need to use robust()
> (from metafor) or coef_test() (from clubSandwich) (or the robumeta package)
> if you want RVE.
> >
> > Best,
> > Wolfgang
> >
> >> -----Original Message-----
> >> From: Tarun Khanna [mailto:khanna using hertie-school.org]
> >> Sent: Thursday, 23 April, 2020 12:49
> >> To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> >> Subject: Re: [R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11
> >>
> >> Thank you for your reply Wolfgang.
> >>
> >> 1. I am refering to a book by Ringquist (2013) on "Meta analysis in
> Public
> >> Policy". In that book he mentions that "while REML estimate of tau2 is
> >> assymptotically unbiased and efficient, it does have three shortcomings.
> >> First, the beneefits of REML only occur is large samples. In smaller
> >> samples, other methods of estimating tau2 are preferred." He further
> >> discusses moments estimate as an alternate to REML.
> >>
> >> 2. With regards to clustering, I am using the rma.mv function with
> >> specification for inner (effect ID) and outer variable (study ID). In my
> >> data, there are several estimates from the same studies which are of
> course
> >> then correlated.  I understood that this would be equivalent to using
> what
> >> the book calles clustered robust variance estimation or VRE. In this
> context
> >> Ringquist mentions that "White's (1980) original covariance matrix is
> only
> >> consistent, not unbiased, which means that we can have confidence in
> >> parameter standard errors from this martix only in large samples."
> >>
> >> He further states that "Under H0: bj = 0, the ratio bj/sbj i ~ N(0,1)
> and
> >> therefore Wald test for H0 will be a a Z score. While this is true
> >> asymptotically, with small numbes of clusters this assumption does not
> hold
> >> and hypotesis tests using Z scores will return inflated type I errors
> >> (Cameron, Gelbach, and Miller 2008)."
> >>
> >> The book uses stata to implement these concepts. I am of course trying
> to
> >> find the equivalent in R so I am not 100 percent sure if these are
> >> equivalent. Please do tell me if you think otherwise. Can you also
> refer to
> >> another resource perhaps to understand the theory?
> >>
> >> In your reply you mentioned " rma() with what you get from robust() or
> >> clubSandwich:: coef_test() ". I have not used this. How would this be
> >> different from rma.mv?
> >>
> >> Best
> >> Tarun
> >>
> >> Tarun Khanna
> >> PhD Researcher
> >>
> >> Hertie School
> >>
> >> Friedrichstraße 180
> >> 10117 Berlin ∙ Germany
> >> khanna using hertie-school.org ∙ www.hertie-school.org
> >> ________________________________________
> >> From: Viechtbauer, Wolfgang (SP)
> >> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> >> Sent: 23 April 2020 11:59:45
> >> To: Tarun Khanna; r-sig-meta-analysis using r-project.org
> >> Subject: RE: [R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11
> >>
> >> Dear Tarun,
> >>
> >> Not sure where you are getting the impression from that method-of-moment
> >> estimators might be preferable for small sample sizes.
> >>
> >> With respect to your theoretical question: It's not clear to me which
> >> specific results you are comparing. If you are comparing the fit of a
> >> standard RE/ME model from rma() with what you get from robust() or
> >> clubSandwich:: coef_test() where you specify some higher-order
> clustering
> >> variable, then an increase in the SEs is expected when there is indeed
> >> dependency within clusters. And that's as it should be. You are then in
> fact
> >> preventing too many Type I errors, so it's the other way around.
> >>
> >> Also, in the future, please use a more meaningful subject than
> "[R-meta] R-
> >> sig-meta-analysis Digest, Vol 35, Issue 11" (I won't change it now as
> this
> >> might break the threading in email clients).
> >>
> >> Best,
> >> Wolfgang
> >>
> >>> -----Original Message-----
> >>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
> >> project.org]
> >>> On Behalf Of Tarun Khanna
> >>> Sent: Tuesday, 21 April, 2020 16:44
> >>> To: James Pustejovsky
> >>> Cc: r-sig-meta-analysis using r-project.org
> >>> Subject: Re: [R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11
> >>>
> >>> Thank you, James. I understand that the moments estimators might be
> more
> >>> useful when working with small sample sizes. In my data set I have
> about 60
> >>> studies and 110 effect sizes. So as such the dataset is not small. But
> I do
> >>> want to estimate effect sizes for smaller sets of the data (there are
> >>> multiple set of interventions which can be distinguished). In the
> smaller
> >>> sets the number of effects decreases to as low as 15-30. In this
> context, I
> >>> thought DL might be a better estimator. I will look into the robumeta
> >>> package.
> >>>
> >>> I also have a theoretical question  around RVE. The estimates that I
> get
> >> for
> >>> RVE have much higher standard errors compared to the DL/REML
> estimator. I
> >>> understand that this is to be expected, RVE is also likely to result in
> >>> higher Type I errors. Is there any way to control for that in the
> metafor
> >>> package?
> >>>
> >>> Best
> >>> Tarun
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]