[R-meta] Multilevel meta-analysis using z-scores

[Viechtbauer, Wolfgang (SP)]

Ah ok, clear.

Well, I have never seen a meta-analysis of such test statistics. For one, test statistics are an increasing function of sample size. If the null hypothesis is false, then larger sample sizes will tend to yield larger test statistics. Standard effect size measures used for meta-analyses are specifically not constructed this way; they should be consistent, that is, they should converge to the true effect/outcome as the sample size increases, but not increase in size.

Also, computing the sampling variance of such test statistics can be difficult. This is trivial for a z-statistic (it is always 1 -- although in most cases the z-statistics we compute are just large-sample approximations and do not truly have a variance of 1), but for other statistics things are more difficult. For example, for a t-statistic, the variance is df/(df-2) if the null hypothesis is true, but if not, then the variance depends on the non-centrality parameter.

There is an approach that comes from the economics literature, called 'meta-significance testing', that actually makes use of the fact that test statistics will tend to increase with larger sample sizes when the null hypothesis is false. So, if we want to test whether there really is an effect, we can regress the test statistics on the sample sizes (or functions thereof) and if the slope of such a regression model is significantly different from zero, then this is evidence that the null is false. This doesn't give you an estimate of the size of the effect, so it is just a method for testing whether a 'genuine' effect is present or not. Here is an article describing this idea:

Stanley, T. D. (2008). Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70(1), 103-127.

But that's not what you seem to have in mind, as you want to conduct a 'regular' meta-analysis. In that case, you will have to develop new methodology and convince your readers that what you are doing in sound.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of BARANYI Gergö
Sent: Tuesday, 22 May, 2018 17:46
To: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Multilevel meta-analysis using z-scores

Hi Wolfgang,
 
For continuous outcomes, I have divided the beta coefficient with the standard error. 
For any ratios (OR, HR, RR) I have used the following R codes:
 
OR <- 1.12
OR_low <-1
LogOR <- log(OR)
LogOR_low <-log(OR_low)
diff <- LogOR - LogOR_low
seor <- diff / 1.96
z <- LogOR / seor
 
As there were also some experimental studies, if not reported, I calculated the t-values from the summary statistics by testing the proportions of individuals in each groups: 
 
prop.test(c(11,7), c(28,28))
 
Many thanks,
Gergő