[R-meta] Question Re: CI in metafor

[Viechtbauer, Wolfgang (SP)]

Dear Caitlin,

Please cc the mailing list.

I would say this is a false dichotomy. Take a standard RE model. By default, the SE of mu-hat in metafor is computed with:

SE(mu-hat) = sqrt(1/sum(wi))

where wi = 1/(vi + tau^2).

So the SE is not just computed based on sampling error variance (vi), but also taking the between-study heterogeneity into consideration (tau^2).

In the 'barebones' H&S method (with correlations), the SE is computed with:

SE(mu-hat) = sqrt(sum(ni*(ri - mu-hat)^2)/(sum(ni) * k))

where mu-hat = sum(ni*ri)/sum(ni). The variability of the ri values around mu-hat is a function of the sampling variances and the between-study heterogeneity, so this formula also takes these two components into consideration.

So, conceptually, both equations are doing the same thing.

Best,
Wolfgang

-----Original Message-----
From: Caitlin Porter [mailto:cporter2 using central.uh.edu] 
Sent: Saturday, 02 June, 2018 0:26
To: Viechtbauer, Wolfgang (SP)
Subject: Re: [R-meta] Question Re: CI in metafor

Thank you for your reply! And thank you for clarifying that the HS terminology is different from statisticians/methodologists' terminology. 

To clarify, my understanding is that HS recommend creating CIs using the standard error based on the standard deviation of corrected effect sizes across studies (divided by the square root of the number of studies), whereas the approach taken in metafor creates CIs using the standard error based on the standard deviation of sampling error variance across studies (divided by the square root of the number of studies). I hope this doesn't sound too convoluted...

On Wed, May 30, 2018 at 9:03 AM, Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Dear Caitlin,

I do not have a copy of S&H (2013), only the 2004 (i.e., 2nd) edition. So I cannot look up what exactly you mean by "square root of the variance of the corrected Rs/square root of k". Note that the notation and terminology used by H&S differs quite a bit from how statisticians/methodologists outside of the 'psychometric meta-analysis world' talk about these models/methods. Hence, I am also not entirely sure what you mean by "square root of Ave(ve)/square root of k".

The methods as implemented in metafor are based on mixed-effects models and use standard estimation methods (weighted least squares, maximum likelihood estimation, etc.). Hence, the calculations are just how things are calculated under such models and estimation methods.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Caitlin Porter
Sent: Tuesday, 29 May, 2018 22:17
To: r-sig-meta-analysis using r-project.org
Subject: [R-meta] Question Re: CI in metafor

Recently, I have been using the metafor program to run analyses for a
meta-analysis based on the Hunter-Schmidt approach. I have been
double-checking my own calculations with the program, and I found that my
calculations of confidence intervals do not align with the calculations
provided by metafor.

I am using the SD of the corrected Rs as a basis for my CIs (square root of
the variance of the corrected Rs/square root of k), as noted in Schmidt and
Hunter (2013, p. 154, pp. 420-421). However, the confidence intervals that
appear in the metafor output appear to be based upon the sampling error
variance (the square root of Ave(ve)/square root of k).

1) Is there was a reason that the CIs in metafor are based upon this value
as opposed to the one suggested by HS?
2) Is one approach preferred over the other?

Thank you in advance for your insights on this matter!

-- 
Caitlin M. Porter, Ph.D.

Assistant Professor
Industrial and Organizational Psychology
Department of Psychology
University of Houston
3625 Cullen Blvd.
Houston, TX 77204