[R-meta] Questions about Omnibus tests
Rafael Rios
bior@f@elrm @ending from gm@il@com
Sun Nov 4 15:08:24 CET 2018
Dear Michael,
Thanks for the advise and the reference. I did not propose to permanently
delete outliers from the analysis. This approach is only useful to evaluate
if I can obtain the same conclusions with and without the outliers, but
both results should be showed in the paper. Alternatively, I don't know if
I can use a sensitivity analysis for rma.mv function to test the
consistency of the results. I am concerned about to draw biased conclusions
on differences or similarities in average effect size between subgroups.
Best wishes,
Rafael.
__________________________________________________________
Dr. Rafael Rios Moura
*scientia amabilis*
Behavioral Ecologist, PhD
Postdoctoral Researcher
Universidade Estadual de Campinas (UNICAMP)
Campinas, São Paulo, Brazil
Currículo Lattes: http://lattes.cnpq.br/4264357546465157
ORCID: http://orcid.org/0000-0002-7911-4734
Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
<http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8>
Em dom, 4 de nov de 2018 às 11:43, Michael Dewey <lists using dewey.myzen.co.uk>
escreveu:
> Dear Rafael
>
> I will let Wolfgang respond about the details of models but as far as
> heterogeneity is concerned.
>
> Eliminating outliers without concrete reasons may be removing the most
> interesting observations.
>
> If the primary studies are well estimated then heterogeneity is bound to
> be large. See for example Rücker, G, Schwarzer, G, Carpenter, J R and
> Schumacher, M, Undue reliance on I^2 in assessing heterogeneity may
> mislead BMC Medical Research Methodology 2008 (8) 79
>
> It is really the pattern of heterogeneity which you need to examine in
> your funnel plot and then try to explain it.
>
> Michael
>
> On 03/11/2018 23:31, Rafael Rios wrote:
> > Dear Wolfgang,
> >
> > Could you please help me again with new questions?
> >
> > Should I build model1 rather than model2 to control for the dependency
> > among studyID and effectsizeID?
> >
> > model1=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice,
> > random=list(~1|studyID/effectsizeID, ~1|species1), data = h_mc)
> > model2=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice,
> > random=list(~1|effectsizeID, ~1|studyID, ~1|species1), data = h_mc)
> >
> > I used your script to calculate I² and found a high heterogeneity in my
> > model (86.63%).
> > #I²:
> http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate
> > W <- diag(1/h_mc$vzf)
> > X <- model.matrix(model1)
> > P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
> > 100 * sum(meta$sigma2) / (sum(meta$sigma2) +
> (meta$k-meta$p)/sum(diag(P)))
> >
> > Do you have suggestions on how to handle with high heterogeneity among
> > effect sizes? How may I conduct sensitivity tests in a multilevel
> > meta-analysis using metafor? I identified (using a funnel plot) and
> > removed outliers to reduce the heterogeneity and redo the model. Is this
> > approach suitable to evaluate potential bias in results? Or are there
> > better alternatives?
> >
> > Best wishes,
> >
> > Rafael.
> > __________________________________________________________
> >
> > Dr. Rafael Rios Moura
> > /scientia amabilis/
> >
> > Behavioral Ecologist, PhD
> > Postdoctoral Researcher
> > Universidade Estadual de Campinas (UNICAMP)
> > Campinas, São Paulo, Brazil
> >
> > Currículo Lattes: http://lattes.cnpq.br/4264357546465157
> > ORCID: http://orcid.org/0000-0002-7911-4734
> > Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
> >
> >
> >
> >
> > <http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8>
> >
> >
> > Em ter, 30 de out de 2018 às 16:01, Rafael Rios <biorafaelrm using gmail.com
> > <mailto:biorafaelrm using gmail.com>> escreveu:
> >
> > Dear Wolfgang,
> >
> > Thank you for the amazing clarifications! I think I finally have a
> > better picture about the meta-analytic procedures.
> >
> > Best wishes,
> >
> > Rafael.
> > __________________________________________________________
> >
> > Dr. Rafael Rios Moura
> > /scientia amabilis/
> >
> > Behavioral Ecologist, PhD
> > Postdoctoral Researcher
> > Universidade Estadual de Campinas (UNICAMP)
> > Campinas, São Paulo, Brazil
> >
> > Currículo Lattes: http://lattes.cnpq.br/4264357546465157
> > ORCID: http://orcid.org/0000-0002-7911-4734
> > Research Gate:
> https://www.researchgate.net/profile/Rafael_Rios_Moura2
> >
> >
> >
> >
> > <
> http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8>
> >
> >
> > Em ter, 30 de out de 2018 às 15:28, Viechtbauer, Wolfgang (SP)
> > <wolfgang.viechtbauer using maastrichtuniversity.nl
> > <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>> escreveu:
> >
> > Dear Rafael,
> >
> > 1. "Does the QM-test, with an intercept in the model, evaluates
> > if the average true outcomes of subgroups differ from the
> > reference level or from 0?"
> >
> > >From the reference level.
> >
> > "Why are the graph results so different from the QM-test with an
> > intercept in the model?"
> >
> > Your graph is not correct. It should be:
> >
> > preds <- predict(meta, newmods=rbind(c(0,0), c(1,0), c(0,1)))
> > forest(preds$pred, sei=preds$se, slab=c("female", "male",
> "mutual"))
> >
> > The differences between the three levels are small.
> >
> > "Should I evaluate results using anova(meta,btt=1:3)?"
> >
> > anova(meta,btt=1:3) tests if all 3 groups have a zero effect.
> > That does not test for differences between groups.
> >
> > "Was the argument linfct=rbind(c(0,0,1)) used to compare the
> > subgroups of female choice (reference level) and male choice?"
> >
> > No, this compares 'mutual' with 'female'.
> >
> > "What am I evaluating by using summary(glht(meta,
> > linfct=rbind(female=c(1,0,0), male=c(0,1,0))), test=Chisqtest())"
> >
> > You are evaluating whether the intercept (and hence the effect
> > for 'female') is 0 and whether there is a difference between
> > 'male' and 'female'.
> >
> > 2. "What is the best approach to measure heterogeneity in a
> > multilevel meta-analysis?"
> >
> > I don't know what is best. The link you posted provides some
> > possibilities for computing I^2-like measures for
> > multilevel/multivariate models.
> >
> > 3. "I used the standard deviation to weight the effect sizes,
> > according to Zaykin (2011). Is variance a better measure of
> > weight than se in a multilevel meta-analysis?"
> >
> > As mentioned by Michael, this article is irrelevant.
> >
> > 4. "An alternative could be to include this potential_sce as a
> > fixed variable."
> >
> > Sure.
> >
> > "Is this model more appropriate?: meta=rma.mv
> > <http://rma.mv>(zf, sezf, mods=~mate_choice+potential_sce,
> > random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data =
> > h_mc)"
> >
> > You should pass the variances to the function:
> >
> > meta=rma.mv <http://rma.mv>(zf, vzf,
> > mods=~mate_choice+potential_sce, random = list (~1|effectsizeID,
> > ~1|studyID, ~1|species1), data = h_mc)
> >
> > Best,
> > Wolfgang
> >
> > -----Original Message-----
> > From: Rafael Rios [mailto:biorafaelrm using gmail.com
> > <mailto:biorafaelrm using gmail.com>]
> > Sent: Tuesday, 30 October, 2018 6:16
> > To: Viechtbauer, Wolfgang (SP)
> > Cc: Michael Dewey; r-sig-meta-analysis using r-project.org
> > <mailto:r-sig-meta-analysis using r-project.org>
> > Subject: Re: [R-meta] Questions about Omnibus tests
> >
> > Dear Wolfgang,
> >
> > Thank you for the very helpful advices! I will be grateful if
> > you could help me again with my new questions. I organized them
> > in the topics bellow.
> >
> > 1. Does the QM-test, with an intercept in the model, evaluates
> > if the average true outcomes of subgroups differ from the
> > reference level or from 0? I found a p>0.05, probably meaning
> > that there is no difference among subgroups. However, if you
> > analyze the graph, there a higher effect size for the subgroup
> > of female choice compared to others. So, I am not sure about the
> > best approach to evaluate differences among outcomes. Why are
> > the graph results so different from the QM-test with an
> > intercept in the model? Should I evaluate results using
> > anova(meta,btt=1:3)?
> >
> > You also suggested that the script for pairwise comparisons was
> > wrong. According to the link that you provided, it can also be
> > drawn as summary(glht(meta, linfct=rbind(c(0,0,1), c(0,1,0),
> > c(0,-1,1))), test=adjusted("none")). Was the argument
> > linfct=rbind(c(0,0,1)) used to compare the subgroups of female
> > choice (reference level) and male choice? What am I evaluating
> > by using summary(glht(meta, linfct=rbind(female=c(1,0,0),
> > male=c(0,1,0))), test=Chisqtest())?
> >
> > 2. Thank you for the correction of I² formula. What is the best
> > approach to measure heterogeneity in a multilevel meta-analysis?
> > Maybe, this one:
> >
> http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate
> >
> > 3. I used the standard deviation to weight the effect sizes,
> > according to Zaykin (2011). Is variance a better measure of
> > weight than se in a multilevel meta-analysis? Reference: D. V.
> > Zaykin, Optimally weighted Z-test is a powerful method for
> > combining probabilities in meta-analysis. J. Evol. Biol. 24,
> > 1836–1841 (2011).
> >
> > 4. Finally, I agree with the exclusion of potential_sce as a
> > random variable. However, I need to control for this variable.
> > An alternative could be to include this potential_sce as a fixed
> > variable. Is this model more appropriate?: meta=rma.mv
> > <http://rma.mv>(zf, sezf, mods=~mate_choice+potential_sce,
> > random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data =
> > h_mc).
> >
> > Thank you again for the help.
> >
> > Best wishes,
> >
> > Rafael.
> > __________________________________________________________
> >
> > Dr. Rafael Rios Moura
> > scientia amabilis
> >
> > Behavioral Ecologist, PhD
> > Postdoctoral Researcher
> > Universidade Estadual de Campinas (UNICAMP)
> > Campinas, São Paulo, Brazil
> >
> > Currículo Lattes: http://lattes.cnpq.br/4264357546465157
> > ORCID: http://orcid.org/0000-0002-7911-4734
> > Research Gate:
> > https://www.researchgate.net/profile/Rafael_Rios_Moura2
> >
>
> --
> Michael
> http://www.dewey.myzen.co.uk/home.html
>
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list