[R] Mean effect size in meta-analysis using Metafor

[Viechtbauer Wolfgang (STAT)]

Try this:

When fitting the model based on the subset data, fix the variance components to the values from the model with the categorical moderator. So, in your rma.mv() call, use:

sigma2=c(0.065, 0.113)

(ideally, use values that are less rounded). The overall effect size estimate from this model should then be identical to the intercept from the model with the categorical moderator, indicating that the discrepancy is exactly due to the fact described under the link provided by Michael.

If not, there is something else going on.

Best,
Wolfgang

-- 
Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and    
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD    
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com    

> -----Original Message-----
> From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of Carlijn .
> Sent: Monday, December 21, 2015 14:35
> To: Michael Dewey; r-help at r-project.org
> Subject: Re: [R] Mean effect size in meta-analysis using Metafor
> 
> Dear Michael,
> 
> Thanks for your reaction. The estimates in the example in the link are
> not exactly identical, but almost. However, in my case, there is a
> substantial difference between the estimates of the overall effect.
> 
> For example: I want to estimate the effect size of different mental
> health disorders. I start with estimating the effect size of substance
> use disorder (SUD), using a three-level random effects model.
> 
> In the first approach I have fitted an intercept only model with a subset
> of the data including only data on SUD. The overall effect size of SUD is
> d = 0.185 (SE = 0.085), p = .033.
> 
> Using the second approach, I have included a categorical moderator
> 'disorder' (SUD, DBD, ADHD). The reference group is SUD and I have added
> the predictors DBD (i.e., DBD is coded with '1', and SUD and ADHD with
> '0') and ADHD (i.e., ADHD is coded with '1' and SUD and DBD with '0').
> The mean effect size of SUD (intercept) is d = 0.300 (SE = 0.104), p =
> .005.
> 
> To conclude, there is a substantial difference between the estimated
> effect sizes (d = 0.185 versus d = 0.300).
> 
> Approach 1:
> > summary(sud, digits=3)
> Multivariate Meta-Analysis Model (k = 49; method: REML)
>   logLik  Deviance       AIC       BIC      AICc
>  -11.808    23.616    29.616    35.230    30.161
> 
> Variance Components:
>            estim   sqrt  nlvls  fixed  factor
> sigma^2.1  0.042  0.206     49     no       y
> sigma^2.2  0.050  0.223     13     no      ID
> 
> Test for Heterogeneity:
> Q(df = 48) = 409.874, p-val < .001
> 
> Model Results:
> 
> estimate       se     tval     pval    ci.lb    ci.ub
>    0.185    0.085    2.189    0.033    0.015    0.356        *
> 
> Approach 2:
> > summary(external, digits=3)
> Multivariate Meta-Analysis Model (k = 123; method: REML)
>   logLik  Deviance       AIC       BIC      AICc
>  -58.470   116.940   126.940   140.878   127.467
> 
> Variance Components:
>            estim   sqrt  nlvls  fixed  factor
> sigma^2.1  0.065  0.256    123     no       y
> sigma^2.2  0.113  0.336     17     no      ID
> 
> Test for Residual Heterogeneity:
> QE(df = 120) = 846.602, p-val < .001
> 
> Test of Moderators (coefficient(s) 2,3):
> QM(df = 2) = 0.956, p-val = 0.387
> 
> Model Results:
> 
>           estimate          se      tval     pval    ci.lb  ci.ub
> intrcpt      0.300       0.104     2.874    0.005    0.093  0.506  **
> DBD          0.114       0.084     1.354    0.178   -0.053  0.281
> ADHD         0.087       0.104     0.836    0.405   -0.119  0.293
> 
> > Subject: Re: [R] Mean effect size in meta-analysis using Metafor
> > To: wibbeltjec at hotmail.com; r-help at r-project.org
> > From: lists at dewey.myzen.co.uk
> > Date: Sat, 12 Dec 2015 16:12:58 +0000
> >
> > Dear Carlijn
> >
> > I wonder whether
> >
> > http://www.metafor-
> project.org/doku.php/tips:comp_two_independent_estimates
> >
> > answers your question? If you had given us an example of your fitting
> > procedure we might know for sure.
> >
> > On 12/12/2015 15:35, Carlijn . wrote:
> > >
> > >
> > > Hi all,
> > >
> > > I have a question about doing a meta-analysis, in particular a three-
> level
> > > meta-analysis using Metafor.
> > >
> > > I have estimated the mean overall effect size of males by using two
> different
> > > ways:
> > >
> > > 1. moderator analysis (male = 0, female = 1) using the whole data set
> > >
> > > 2. intercept-only model with a subset of the data (only males)
> > >
> > > The mean effect size estimated by using the categorical moderator
> analysis
> > > (1) differs considerably from the overall mean effect size estimated
> in an
> > > intercept-only model using a subset of the data (2).
> > >
> > > Can someone explain this? Which method gives a better estimation of
> the
> > > effect?
> > >
> > > Thank you in advance!