[R-meta] Some questions: effect sizes, heterogeneity and Egger's test

[James Pustejovsky]

Hi Teresa,

Responses below.

James

On Mon, Sep 13, 2021 at 9:54 AM Teresa Luther <Teresa.Luther using gmx.de> wrote:

> Dear All,
>
> I conducted several meta-analyses. I used the R package "metafor" as well
> as the free open-source software OpenMetaAnalyst for conducting the
> analyses.
>
> Now, I am writing up the results and face some uncertainty with regard to
> the statistics.
>
> 1) I have a p-value of .000 for several effect sizes (Hedge's g). Also for
> Higgins' I^2 (heterogeneity) I sometimes get this value for p.
> I would tend to write p < .001 and possibly footnote the .000. Is this
> possible or what would you suggest in such a case? Simply report it as p <
> .001?
>

Yes, reporting as p < .001 is sensible.


>
> 2) As a measure of heterogeneity, I interpret Higgins' I^2. I interpret
> the values above 25% as low heterogeneity (following Higgins et al. 2003).
> In some of the analyses, I get a value of 0%. I would now have to write
> that there is no heterogeneity. However, I consider this value almost
> impossible, since there is always a certain variance between the studies.
> Since I had assumed heterogeneity between the studies, I had also performed
> the calculations on the basis of a random-effects model.
> I am also not sure whether values of 25, 50 and 75 % have to be considered
> as cut-off values or whether values in between can also be interpreted as
> for example "small to medium" for I^2=45 %.
> This question also arises for me with Hedge's g.
>

I would recommend reporting and focusing your interpretation on the
estimate of tau, the between-study heterogeneity parameter, rather than on
I^2. The interpretation of I^2 depends on the distribution of sample sizes
in your primary studies, so it doesn't make much sense at all to use
decontextualized benchmarks based on I^2. For a more detailed explanation
of this reasoning and some additional suggestions about how to interpret
heterogeneity, check out the following article:

Borenstein, M., Higgins, J. P., Hedges, L. V., & Rothstein, H. R. (2017).
Basics of meta‐analysis: I2 is not an absolute measure of
heterogeneity. *Research
synthesis methods*, *8*(1), 5-18.


>
> 3) To investigate a possible publication bias, I ran Egger's regression
> test in R for some of the analyses. I would like to report the regression
> intercept as well and assume that the intercept is the “b“ (provided
> together with CI) I get in the output. Would it be correct to report this
> value as beta^ with index 1? In the literature I found that in most
> meta-analyses only report the p-value for Egger's test, however the
> recommendation is made to report the intercept as well. Now I would like to
> report this regression intercept correctly.
>
> I don't think there is a standard symbol or notation for the intercept
from Egger's regression test. The more important thing is to give an
accurate description of the coefficient, as the estimated average effect
size in a study with zero sampling error, based on a model that includes
the standard error as a linear predictor. In recent versions of metafor,
the regtest() function reports this estimate and describes it as the "Limit
Estimate (as sei -> 0)".

	[[alternative HTML version deleted]]