[R-meta] Questions about Omnibus tests
Michael Dewey
li@t@ @ending from dewey@myzen@co@uk
Sun Nov 4 14:43:00 CET 2018
Dear Rafael
I will let Wolfgang respond about the details of models but as far as
heterogeneity is concerned.
Eliminating outliers without concrete reasons may be removing the most
interesting observations.
If the primary studies are well estimated then heterogeneity is bound to
be large. See for example Rücker, G, Schwarzer, G, Carpenter, J R and
Schumacher, M, Undue reliance on I^2 in assessing heterogeneity may
mislead BMC Medical Research Methodology 2008 (8) 79
It is really the pattern of heterogeneity which you need to examine in
your funnel plot and then try to explain it.
Michael
On 03/11/2018 23:31, Rafael Rios wrote:
> Dear Wolfgang,
>
> Could you please help me again with new questions?
>
> Should I build model1 rather than model2 to control for the dependency
> among studyID and effectsizeID?
>
> model1=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice,
> random=list(~1|studyID/effectsizeID, ~1|species1), data = h_mc)
> model2=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice,
> random=list(~1|effectsizeID, ~1|studyID, ~1|species1), data = h_mc)
>
> I used your script to calculate I² and found a high heterogeneity in my
> model (86.63%).
> #I²: http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate
> W <- diag(1/h_mc$vzf)
> X <- model.matrix(model1)
> P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
> 100 * sum(meta$sigma2) / (sum(meta$sigma2) + (meta$k-meta$p)/sum(diag(P)))
>
> Do you have suggestions on how to handle with high heterogeneity among
> effect sizes? How may I conduct sensitivity tests in a multilevel
> meta-analysis using metafor? I identified (using a funnel plot) and
> removed outliers to reduce the heterogeneity and redo the model. Is this
> approach suitable to evaluate potential bias in results? Or are there
> better alternatives?
>
> Best wishes,
>
> Rafael.
> __________________________________________________________
>
> Dr. Rafael Rios Moura
> /scientia amabilis/
>
> Behavioral Ecologist, PhD
> Postdoctoral Researcher
> Universidade Estadual de Campinas (UNICAMP)
> Campinas, São Paulo, Brazil
>
> Currículo Lattes: http://lattes.cnpq.br/4264357546465157
> ORCID: http://orcid.org/0000-0002-7911-4734
> Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
>
>
>
>
> <http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8>
>
>
> Em ter, 30 de out de 2018 às 16:01, Rafael Rios <biorafaelrm using gmail.com
> <mailto:biorafaelrm using gmail.com>> escreveu:
>
> Dear Wolfgang,
>
> Thank you for the amazing clarifications! I think I finally have a
> better picture about the meta-analytic procedures.
>
> Best wishes,
>
> Rafael.
> __________________________________________________________
>
> Dr. Rafael Rios Moura
> /scientia amabilis/
>
> Behavioral Ecologist, PhD
> Postdoctoral Researcher
> Universidade Estadual de Campinas (UNICAMP)
> Campinas, São Paulo, Brazil
>
> Currículo Lattes: http://lattes.cnpq.br/4264357546465157
> ORCID: http://orcid.org/0000-0002-7911-4734
> Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
>
>
>
>
> <http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8>
>
>
> Em ter, 30 de out de 2018 às 15:28, Viechtbauer, Wolfgang (SP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl
> <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>> escreveu:
>
> Dear Rafael,
>
> 1. "Does the QM-test, with an intercept in the model, evaluates
> if the average true outcomes of subgroups differ from the
> reference level or from 0?"
>
> >From the reference level.
>
> "Why are the graph results so different from the QM-test with an
> intercept in the model?"
>
> Your graph is not correct. It should be:
>
> preds <- predict(meta, newmods=rbind(c(0,0), c(1,0), c(0,1)))
> forest(preds$pred, sei=preds$se, slab=c("female", "male", "mutual"))
>
> The differences between the three levels are small.
>
> "Should I evaluate results using anova(meta,btt=1:3)?"
>
> anova(meta,btt=1:3) tests if all 3 groups have a zero effect.
> That does not test for differences between groups.
>
> "Was the argument linfct=rbind(c(0,0,1)) used to compare the
> subgroups of female choice (reference level) and male choice?"
>
> No, this compares 'mutual' with 'female'.
>
> "What am I evaluating by using summary(glht(meta,
> linfct=rbind(female=c(1,0,0), male=c(0,1,0))), test=Chisqtest())"
>
> You are evaluating whether the intercept (and hence the effect
> for 'female') is 0 and whether there is a difference between
> 'male' and 'female'.
>
> 2. "What is the best approach to measure heterogeneity in a
> multilevel meta-analysis?"
>
> I don't know what is best. The link you posted provides some
> possibilities for computing I^2-like measures for
> multilevel/multivariate models.
>
> 3. "I used the standard deviation to weight the effect sizes,
> according to Zaykin (2011). Is variance a better measure of
> weight than se in a multilevel meta-analysis?"
>
> As mentioned by Michael, this article is irrelevant.
>
> 4. "An alternative could be to include this potential_sce as a
> fixed variable."
>
> Sure.
>
> "Is this model more appropriate?: meta=rma.mv
> <http://rma.mv>(zf, sezf, mods=~mate_choice+potential_sce,
> random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data =
> h_mc)"
>
> You should pass the variances to the function:
>
> meta=rma.mv <http://rma.mv>(zf, vzf,
> mods=~mate_choice+potential_sce, random = list (~1|effectsizeID,
> ~1|studyID, ~1|species1), data = h_mc)
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Rafael Rios [mailto:biorafaelrm using gmail.com
> <mailto:biorafaelrm using gmail.com>]
> Sent: Tuesday, 30 October, 2018 6:16
> To: Viechtbauer, Wolfgang (SP)
> Cc: Michael Dewey; r-sig-meta-analysis using r-project.org
> <mailto:r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Questions about Omnibus tests
>
> Dear Wolfgang,
>
> Thank you for the very helpful advices! I will be grateful if
> you could help me again with my new questions. I organized them
> in the topics bellow.
>
> 1. Does the QM-test, with an intercept in the model, evaluates
> if the average true outcomes of subgroups differ from the
> reference level or from 0? I found a p>0.05, probably meaning
> that there is no difference among subgroups. However, if you
> analyze the graph, there a higher effect size for the subgroup
> of female choice compared to others. So, I am not sure about the
> best approach to evaluate differences among outcomes. Why are
> the graph results so different from the QM-test with an
> intercept in the model? Should I evaluate results using
> anova(meta,btt=1:3)?
>
> You also suggested that the script for pairwise comparisons was
> wrong. According to the link that you provided, it can also be
> drawn as summary(glht(meta, linfct=rbind(c(0,0,1), c(0,1,0),
> c(0,-1,1))), test=adjusted("none")). Was the argument
> linfct=rbind(c(0,0,1)) used to compare the subgroups of female
> choice (reference level) and male choice? What am I evaluating
> by using summary(glht(meta, linfct=rbind(female=c(1,0,0),
> male=c(0,1,0))), test=Chisqtest())?
>
> 2. Thank you for the correction of I² formula. What is the best
> approach to measure heterogeneity in a multilevel meta-analysis?
> Maybe, this one:
> http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate
>
> 3. I used the standard deviation to weight the effect sizes,
> according to Zaykin (2011). Is variance a better measure of
> weight than se in a multilevel meta-analysis? Reference: D. V.
> Zaykin, Optimally weighted Z-test is a powerful method for
> combining probabilities in meta-analysis. J. Evol. Biol. 24,
> 1836–1841 (2011).
>
> 4. Finally, I agree with the exclusion of potential_sce as a
> random variable. However, I need to control for this variable.
> An alternative could be to include this potential_sce as a fixed
> variable. Is this model more appropriate?: meta=rma.mv
> <http://rma.mv>(zf, sezf, mods=~mate_choice+potential_sce,
> random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data =
> h_mc).
>
> Thank you again for the help.
>
> Best wishes,
>
> Rafael.
> __________________________________________________________
>
> Dr. Rafael Rios Moura
> scientia amabilis
>
> Behavioral Ecologist, PhD
> Postdoctoral Researcher
> Universidade Estadual de Campinas (UNICAMP)
> Campinas, São Paulo, Brazil
>
> Currículo Lattes: http://lattes.cnpq.br/4264357546465157
> ORCID: http://orcid.org/0000-0002-7911-4734
> Research Gate:
> https://www.researchgate.net/profile/Rafael_Rios_Moura2
>
--
Michael
http://www.dewey.myzen.co.uk/home.html
More information about the R-sig-meta-analysis
mailing list