[R-meta] Why am I getting different means when conducting multilevel meta-analysis with factorial moderator vs. as subgroups?

[Viechtbauer, Wolfgang (SP)]

If you do summary(model), you get information criteria (also fitstats(), AIC(), AIC(..., correct=TRUE), and BIC()).

You can also do a likelihood ratio test with anova(res1, res2) where res1 and res2 are the two (nested) models.

Best,
Wolfgang

-----Original Message-----
From: Maximilian Theisen [mailto:Theisen at stud.uni-heidelberg.de] 
Sent: Thursday, 19 April, 2018 11:56
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Why am I getting different means when conducting multilevel meta-analysis with factorial moderator vs. as subgroups?

Hi,

using Wolfgang's approach helped reduce the discrepancy between the
subgroup model results and the moderator model.
This helped me very much!

Is there such a thing as an information criterion or model fit to help
me decide whether I should use struct="UN" or struct="HCS"?

Best,
Max

Am 18.04.2018 um 18:36 schrieb Viechtbauer, Wolfgang (SP):
> Hi Max,
>
> It's difficult to say without the actual data, but here are a few observations/comments.
>
> First of all, it seems to me that the 'subgroup' model is overparameterized. Notice that there are 7 levels for both studyid and sampleid, so unless these are crossed factors (which I assume they are not), then these two are the same. In fact, the estimated variances for these two random effects are identical (0.1307), so probably the actual variance due to this random effects is getting split between these two (or maybe the optimizer started with these values and got stuck there).
>
> Second, the subgroup models allow for the variance components to differ across tasks, but the 'moderator' model does not. The latter also assumes that the correlation of effects within studies and samples for different tasks can only be positive and that the correlation is the same between all pairs of tasks. That may not be true.
>
> You might try:
>
> multi.task <- rma.mv(yi=g, V = var.g, data=df, random=list(~ factor(task.type) | sampleid, ~ factor(task.type) | studyid), mods=~factor(task.type)-1)
>
> which does allows for a negative correlation. With struct="HCS" you could also allow for different variances across tasks. And with struct="UN" you could also allow for different correlations. But the latter estimates 8 variances and 12 correlations and that might be asking a bit much here.
>
> Since studyid and sampleid have nearly the same number of levels, you might consider dropping one of those levels. Maybe you can get this to converge:
>
> multi.task <- rma.mv(yi=g, V = var.g, data=df, random= ~ factor(task.type) | sampleid, struct="UN", mods=~factor(task.type)-1)
>
> I suspect that things may start to look a bit more like the 'subgroup' models then.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Maximilian Theisen
> Sent: Wednesday, 18 April, 2018 18:09
> To: r-sig-meta-analysis at r-project.org
> Subject: [R-meta] Why am I getting different means when conducting multilevel meta-analysis with factorial moderator vs. as subgroups?
>
> Hello!
>
> I am conducting a multilevel meta-analysis using metafor in R. I have
> effect sizes ("esid") nested within samples ("sampleid") nested within
> publications ("studyid"). I have four subgroups ("task.type").
> The mean effect sizes for each subgroup differ depending on whether I
> use task.type as a moderator or run the rma.mv command for each subgroup
> independently.
>
> This is the code I use with task.type as moderator:
> multi.task <- rma.mv(yi=g, V = var.g, data=df, random=list(~ 1 | esid,
> ~1 | sampleid, ~1 | studyid), mods=~factor(task.type)-1)
>
> Multivariate Meta-Analysis Model (k = 142; method: REML)
>
> Variance Components:
>
>             estim    sqrt  nlvls  fixed    factor
> sigma^2.1  0.0942  0.3069    142     no      esid
> sigma^2.2  0.7769  0.8814     29     no  sampleid
> sigma^2.3  0.0000  0.0001     25     no   studyid
>
> Test for Residual Heterogeneity:
> QE(df = 138) = 950.2971, p-val < .0001
>
> Test of Moderators (coefficient(s) 1:4):
> QM(df = 4) = 29.9283, p-val < .0001
>
> Model Results:
>
>                estimate      se     zval    pval    ci.lb    ci.ub   
> factor(task)1    0.6072  0.2360   2.5729  0.0101   0.1446   1.0697   *
> factor(task)2   -0.5173  0.2559  -2.0212  0.0433  -1.0189  -0.0157   *
> factor(task)3    0.5755  0.2048   2.8100  0.0050   0.1741   0.9769  **
> factor(task)4    0.6173  0.4333   1.4246  0.1543  -0.2320   1.4665   
>
> This is the one I use when computing the model for each task.type
> individually:
> task.X <- rma.mv(yi=g, V = var.g, data=df, subset=(task=="X"),
> random=list(~ 1 | esid, ~ 1 |sampleid, ~ 1 | studyid))
>
> Task 1:
>
> Multivariate Meta-Analysis Model (k = 27; method: REML)
>
> Variance Components:
>
>             estim    sqrt  nlvls  fixed    factor
> sigma^2.1  0.1685  0.4105     27     no      esid
> sigma^2.2  0.1307  0.3616      7     no  sampleid
> sigma^2.3  0.1307  0.3616      7     no   studyid
>
> Test for Heterogeneity:
> Q(df = 26) = 115.1759, p-val < .0001
>
> Model Results:
>
> estimate      se     zval    pval    ci.lb   ci.ub  
>  -0.0649  0.2289  -0.2836  0.7767  -0.5135  0.3837  
>
> Task 2:
> estimate      se    zval    pval    ci.lb   ci.ub  
>   0.3374  0.6983  0.4832  0.6290  -1.0312  1.7060  
>
> Task 3:
> estimate      se    zval    pval   ci.lb   ci.ub   
>   0.3862  0.1254  3.0808  0.0021  0.1405  0.6319  **
>
> Task 4:
> estimate      se    zval    pval    ci.lb   ci.ub  
>   0.6126  0.3409  1.7971  0.0723  -0.0555  1.2807  .
>
> Why are the results so different?
>
> Best,
> Max