[R-meta] Within-study variance in a multilevel meta-analytic model

[Viechtbauer, Wolfgang (NP)]

Dear Tharaka,

The reviewer appears to be unfamiliar with meta-analysis in general. This has nothing to do with heteroscedasticity, but with accounting for variance/heterogeneity (between studies and effect sizes within studies in your case). In fact, in the standard random-effects model, we also add an observation-level (i.e., effect-size level) random effect, which is the whole point of the random-effects model (to allow for heterogeneity in the true effects).

For people unfamiliar with meta-analysis, this can be confusing, since in a standard linear regression model (or any standard mixed-effects model), such a random effect would be completely confounded with the error term and hence not identifiable. However, in meta-analysis, we do not have a standard error term whose variance we estimate, but we already have the pre-computed sampling variances of the estimates (which are indeed heteroscedastic, because the estimates will have different sampling variances).

Best,
Wolfgang

> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Tharaka S. Priyadarshana via R-sig-meta-analysis
> Sent: Saturday, December 30, 2023 16:25
> To: r-sig-meta-analysis using r-project.org
> Cc: Tharaka S. Priyadarshana <tharakas.priyadarshana using gmail.com>
> Subject: [R-meta] Within-study variance in a multilevel meta-analytic model
>
> Dear all,
>
> In my meta-analysis, I have multiple effect sizes coming from the same
> study, and these effects are nested with the studies.
>
> So, to account for both between-study and within-study
> variance/heterogeneity, I gave identifiers for each study (StudyID) and
> each effect (EffectSizeID), and included them into the model as random
> effects, i.e., "random = ~1 | (StudyID / EffectSizeID)".
>
> For this, I received a comment from one of the reviewers as follows,
>
> "It took me a long time to figure out why you are using EffectSizeID as a
> random effect, especially since this observation-level random effect
> approach is mainly used to control for overdispersion in Poisson
> regressions. Based on the response to reviewers, I think you are using it
> to account for heteroscedasticity? Focusing on various data sources and
> within-study variability is confusing because each effect size presumable
> comes from just one land cover data source. Framing this as controlling for
> heteroscedasticity would help a lot. If that is not your purpose, please
> edit this paragraph to more clearly explain your approach."
>
> I am a bit unsure how should I respond to this reviewer's comment. If I
> understand correctly, once we define the between and within-study
> variance/heterogeneity
> with random variables, we can get the true effects for each observed effect
> (i.e. each row of the dataset), which then can be used to measure the
> average true effect.
>
> Could someone please let me know whether my interpretation is correct here?
> or am I missing something??
>
> Thank you for your help.
>
> With best wishes,
>
> Tharaka