[R-meta] Categorical mixed effect models and interpretation of results

[Viechtbauer, Wolfgang (SP)]

Hi Alex,

Please keep the mailing list in cc.

I would not say that the results are necessarily 'incorrect'. However, just adding random effects for study assumes that there is no heterogeneity in the outcomes for different experiments within studies, which is a big assumption to make. Also, not adding random effects for species assumes that outcomes for the same species are no more similar to each other than they are when outcomes are measured in different species. Again, that's a questionable assumption. In fact, outcomes in different species may be correlated to the degree that species resemble each other, so that gets us into phylogenetic meta-analyses. See, for example:

Nakagawa, S., & Santos, E. S. A. (2012). Methodological issues and advances in biological meta-analysis. Evolutionary Ecology, 26(5), 1253-1274.

Noble, D. W., Lagisz, M., O'Dea R, E., & Nakagawa, S. (2017). Non-independence and sensitivity analyses in ecological and evolutionary meta-analyses. Molecular Ecology, 26(9), 2410-2425.

Ignoring such issues could lead to underestimation of the uncertainty in parameter estimates and hence inflated Type I error rates.

Best,
Wolfgang

-----Original Message-----
From: Alexander Sullivan (BIO - Student) [mailto:Alexander.J.Sullivan using uea.ac.uk] 
Sent: Tuesday, 19 June, 2018 12:32
To: Viechtbauer, Wolfgang (SP)
Subject: Re: Categorical mixed effect models and interpretation of results

Dear Wolfgang,

Thank you for your prompt reply. This does appear to improve my model.

However, I have noticed in the meta-analytical literature especially within my research area, ocean acidification, that many authors simply use study id as their random effect. Does this mean their results are incorrect? or simply less precise than a more complicated model? 

Best,

Alex

MSc Ecology and Conservation student 
University of East Anglia, UK
________________________________________
From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
Sent: 19 June 2018 09:31:52
To: Alexander Sullivan (BIO - Student); r-sig-meta-analysis using r-project.org
Subject: RE: Categorical mixed effect models and interpretation of results
 
Hi Alex,

Yes, in principle this is right. If 'Treatment' only has two levels, then the QM-test and the test of the treatment coefficient are identical, so you can also just look at the latter.

However, I think your random effects structure is too simple given what you wrote. At the least, it should be something like:

random = ~ 1 | Study_id / Exp_id

where Exp_id is, as the name implies, the experiment id. See:

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011

and esp. the "A Common Mistake in the Three-Level Model" section.

You might also want to consider adding random effects for species. For example:

random = list(~ 1 | Study_id / Exp_id, ~ 1 | Species)

would add species random effects (as a crossed random effect, not nested within study and/or experiment).

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Alexander Sullivan (BIO - Student)
Sent: Tuesday, 19 June, 2018 10:58
To: r-sig-meta-analysis using r-project.org
Subject: [R-meta] Categorical mixed effect models and interpretation of results

Dear all,

In my meta-analysis I have [say] 100 experiments, from 30 studies, across 12 species. These experiments can be equally divided into two treatment levels [low and high]. If I want to determine if there is a significant difference in effect sizes between these two treatment levels would this be the right code:

res <- rma.mv(yi, vi, method = "REML", data = mydata, mods = ~factor(Treatment), random = ~1|Study_id)

... and then from the output of this model if the Qm statistic is significant I could say there is a significant difference in effect sizes between the two treatments?

Thank you for your time,

Alex

MSc Ecology and Conservation student
University of East Anglia, UK