[R-meta] Question about explained variance at 2nd and 3rd level

[Viechtbauer Wolfgang (SP)]

Dear Willemijn,

What you are observing can also happen in 'standard' random/mixed-effects meta-regression models. Here is an example:

set.seed(123123)

k <- 30
vi <- runif(k, .001, 1)
yi <- rnorm(k, 0, sqrt(vi + .2))
xi <- rnorm(k)
rma(yi, vi)
rma(yi ~ xi, vi)

Note that tau^2 actually increases when 'xi' is included as a predictor. Unfortunately, given the nature of these models, this indeed can happen, even without any missing data. Missing data could make the problem worse.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>project.org] On Behalf Of Willemijn van Eldik
>Sent: Monday, 26 February, 2018 12:57
>To: 'r-sig-meta-analysis at r-project.org'
>Subject: [R-meta] Question about explained variance at 2nd and 3rd level
>
>Dear colleagues,
>
>Gratefully, I am making use of your R Package Metafor to conduct a three-
>level meta-analysis. I have a question and I hope that you can help me
>with this question.
>Shortly, I am conducting several meta-analyses for the association
>between the interparental relationship and children’s problem behavior.
>The ES of interest is r (z-transformed) and my dataset concerns 1601 ES
>from 206 independent samples.
>I am conducting several univariate moderator analyses, mostly using
>categorical moderator (dummies).
>
>I would like to extract from the moderator analysis how much variance one
>moderator explains at the 2nd and 3rdlevel. My assumption is that I can
>use the sigma-components, to compare nested models, to calculate this
>amount of explained variance. Following this reasoning, I would have
>expected the sigma-component to become smaller in significant moderator
>models, compared to the overall model (without moderators). However, the
>sigma-components are not consistently becoming smaller when a significant
>moderator is added to the model. Neither are the sigma-components
>consistent in amount, e.g. getting larger, is some models including a
>non-significant moderator.
>Based on these observations of the output, I am hestitant to use the
>sigma-components to  evaluate the explained variance by a moderator on
>each level. My first assumption was that this was caused by missing
>values in moderator analyses. However, this shows even when no missing
>values are present. But maybe I missed something or are mistaken
>somewhere.
>
>My questions are;
>-         -  Is it true that this is caused by missing values and the
>consequence that these models are not comparable with a different N?
>-          - Is there another reason (in case there are no missing
>values)?
>-          - And most important: is there another way of evaluating the
>explained variance by a moderator?
>
>Thanking you in advance,
>
>Best regards,
>Willemijn van Eldik
>​
>PhD Candidate
>
>Erasmus University Rotterdam
>Department of Psychology, Education & Child Studies, ESSB
>Location Mandeville (T) 16, Burg. Oudlaan 50, Room T16-15
>Postbus 1738
>3000 DR  Rotterdam
>E: vaneldik at essb.eur.nl<mailto:vaneldik at essb.eur.nl> T: 0031104089730