[R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11

[Tarun Khanna]

Thank you Wolfgang. Would you recommend any resources that explain in more detail the implementation of RVE in R using the resources that you mentioned? 

Warm regards, 
Tarun Khanna

> On 23. Apr 2020, at 13:01, Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> 
> With respect to 1: I am not aware of any theoretical work or simulation studies that show that DL has better statistical properties than REML in small samples. What studies does Ringquist (2013) cite to support this claim?
> 
> With respect to 2: Indeed, RVE is a method that works in large samples. The issue with treating the ratio of an estimate to its standard error as a z-value applies to pretty much all meta-analytic methods, whether you use RVE or not. Improved methods (such as the K&H adjustment) or small-sample corrections for RVE have been developed that can mitigate this problem. Typically, these methods then lead to using the t-distribution (with some estimated degrees of freedom) as the test statistic.
> 
> And no, just using rma.mv() doesn't do RVE. You need to use robust() (from metafor) or coef_test() (from clubSandwich) (or the robumeta package) if you want RVE.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: Tarun Khanna [mailto:khanna using hertie-school.org]
>> Sent: Thursday, 23 April, 2020 12:49
>> To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
>> Subject: Re: [R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11
>> 
>> Thank you for your reply Wolfgang.
>> 
>> 1. I am refering to a book by Ringquist (2013) on "Meta analysis in Public
>> Policy". In that book he mentions that "while REML estimate of tau2 is
>> assymptotically unbiased and efficient, it does have three shortcomings.
>> First, the beneefits of REML only occur is large samples. In smaller
>> samples, other methods of estimating tau2 are preferred." He further
>> discusses moments estimate as an alternate to REML.
>> 
>> 2. With regards to clustering, I am using the rma.mv function with
>> specification for inner (effect ID) and outer variable (study ID). In my
>> data, there are several estimates from the same studies which are of course
>> then correlated.  I understood that this would be equivalent to using what
>> the book calles clustered robust variance estimation or VRE. In this context
>> Ringquist mentions that "White's (1980) original covariance matrix is only
>> consistent, not unbiased, which means that we can have confidence in
>> parameter standard errors from this martix only in large samples."
>> 
>> He further states that "Under H0: bj = 0, the ratio bj/sbj i ~ N(0,1) and
>> therefore Wald test for H0 will be a a Z score. While this is true
>> asymptotically, with small numbes of clusters this assumption does not hold
>> and hypotesis tests using Z scores will return inflated type I errors
>> (Cameron, Gelbach, and Miller 2008)."
>> 
>> The book uses stata to implement these concepts. I am of course trying to
>> find the equivalent in R so I am not 100 percent sure if these are
>> equivalent. Please do tell me if you think otherwise. Can you also refer to
>> another resource perhaps to understand the theory?
>> 
>> In your reply you mentioned " rma() with what you get from robust() or
>> clubSandwich:: coef_test() ". I have not used this. How would this be
>> different from rma.mv?
>> 
>> Best
>> Tarun
>> 
>> Tarun Khanna
>> PhD Researcher
>> 
>> Hertie School
>> 
>> Friedrichstraße 180
>> 10117 Berlin ∙ Germany
>> khanna using hertie-school.org ∙ www.hertie-school.org
>> ________________________________________
>> From: Viechtbauer, Wolfgang (SP)
>> <wolfgang.viechtbauer using maastrichtuniversity.nl>
>> Sent: 23 April 2020 11:59:45
>> To: Tarun Khanna; r-sig-meta-analysis using r-project.org
>> Subject: RE: [R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11
>> 
>> Dear Tarun,
>> 
>> Not sure where you are getting the impression from that method-of-moment
>> estimators might be preferable for small sample sizes.
>> 
>> With respect to your theoretical question: It's not clear to me which
>> specific results you are comparing. If you are comparing the fit of a
>> standard RE/ME model from rma() with what you get from robust() or
>> clubSandwich:: coef_test() where you specify some higher-order clustering
>> variable, then an increase in the SEs is expected when there is indeed
>> dependency within clusters. And that's as it should be. You are then in fact
>> preventing too many Type I errors, so it's the other way around.
>> 
>> Also, in the future, please use a more meaningful subject than "[R-meta] R-
>> sig-meta-analysis Digest, Vol 35, Issue 11" (I won't change it now as this
>> might break the threading in email clients).
>> 
>> Best,
>> Wolfgang
>> 
>>> -----Original Message-----
>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>> project.org]
>>> On Behalf Of Tarun Khanna
>>> Sent: Tuesday, 21 April, 2020 16:44
>>> To: James Pustejovsky
>>> Cc: r-sig-meta-analysis using r-project.org
>>> Subject: Re: [R-meta] R-sig-meta-analysis Digest, Vol 35, Issue 11
>>> 
>>> Thank you, James. I understand that the moments estimators might be more
>>> useful when working with small sample sizes. In my data set I have about 60
>>> studies and 110 effect sizes. So as such the dataset is not small. But I do
>>> want to estimate effect sizes for smaller sets of the data (there are
>>> multiple set of interventions which can be distinguished). In the smaller
>>> sets the number of effects decreases to as low as 15-30. In this context, I
>>> thought DL might be a better estimator. I will look into the robumeta
>>> package.
>>> 
>>> I also have a theoretical question  around RVE. The estimates that I get
>> for
>>> RVE have much higher standard errors compared to the DL/REML estimator. I
>>> understand that this is to be expected, RVE is also likely to result in
>>> higher Type I errors. Is there any way to control for that in the metafor
>>> package?
>>> 
>>> Best
>>> Tarun