[R-meta] Moderator analysis test of residual heterogeneity confusion

[Viechtbauer, Wolfgang (SP)]

Hi Mia,

Whether analyzing 'one moderator at a time' is a better approach is debatable, but that's a different issue.

So, leaving that issue aside, the results below suggest that all of the heterogeneity (that is not already accounted for by the moderator included in the model) is due to differences between studies (and none of it is due to heterogeneity in the true outcomes within studies). The test for residual heterogeneity even suggests that there is no noteworthy heterogeneity (either within or between studies) left with the moderator included in the model.

Best,
Wolfgang

-----Original Message-----
From: Mia Daucourt [mailto:miadaucourt using gmail.com] 
Sent: Thursday, 19 September, 2019 3:28
To: James Pustejovsky
Cc: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Moderator analysis test of residual heterogeneity confusion

Thank you for the advice! I have added the observation-level random effect and will not run the overfitted model anymore and just stick to one moderator at a time. 

How would I interpret the results of the test for residual heterogeneity for the single moderator below? 

(P.S. this is the only moderator that does not have a significant value for the between-study variance parameter at the observation level and only does at the study level).

Code:

Model2_HMEcomp <- rma.mv(yi, vi., mods = ~ factor(hne_component)-1, random = ~ 1 | study/count, data=Zcalc, test="t")
summary(Model2_HMEcomp)

Partial output:

Multivariate Meta-Analysis Model (k = 456; method: REML)

   logLik   Deviance        AIC        BIC       AICc 
 112.1356  -224.2713  -204.2713  -163.2233  -203.7678   

Variance Components:

            estim    sqrt  nlvls  fixed       factor 
sigma^2.1  0.0136  0.1166     51     no        study 
sigma^2.2  0.0000  0.0000    456     no  study/count 

Test for Residual Heterogeneity:
QE(df = 448) = 409.9810, p-val = 0.9007

Test of Moderators (coefficients 1:8):
F(df1 = 8, df2 = 448) = 6.2947, p-val < .0001

Thanks so much for your guidance!

My best,

Mia