[R-meta] Question about explained variance at 2nd and 3rd level

[Viechtbauer Wolfgang (SP)]

Dear Willemijn,

Please keep the mailing list in cc.

There are other ways of computing R^2-like measures that would not show the same type of 'defect' that you are observing (with the 'proportional reduction in variance components' approach), but the ones that I can think of will only give you an R^2 for the whole model, not each level separately. For example, measures based on the log-likelihood like Nagelkerke's R^2 (which will require using ML estimation, not REML) should always be >= 0, since the log-likelihood cannot decrease with the addition of a predictor.

Best,
Wolfgang

>-----Original Message-----
>From: Willemijn van Eldik [mailto:vaneldik at essb.eur.nl]
>Sent: Tuesday, 27 February, 2018 16:30
>To: Viechtbauer Wolfgang (SP)
>Subject: RE: Question about explained variance at 2nd and 3rd level
>
>Dear Wolfgang,
>
>Thank you for your reply.
>
>And is there (another) way to know or calculate how much variance is
>explained at each level by a certain moderator?
>
>thank you!
>Best,
>Willemijn
>________________________________________
>From: Viechtbauer Wolfgang (SP)
>[wolfgang.viechtbauer at maastrichtuniversity.nl]
>Sent: 26 February 2018 15:53
>To: Willemijn van Eldik; 'r-sig-meta-analysis at r-project.org'
>Subject: RE: Question about explained variance at 2nd and 3rd level
>
>Dear Willemijn,
>
>What you are observing can also happen in 'standard' random/mixed-effects
>meta-regression models. Here is an example:
>
>set.seed(123123)
>
>k <- 30
>vi <- runif(k, .001, 1)
>yi <- rnorm(k, 0, sqrt(vi + .2))
>xi <- rnorm(k)
>rma(yi, vi)
>rma(yi ~ xi, vi)
>
>Note that tau^2 actually increases when 'xi' is included as a predictor.
>Unfortunately, given the nature of these models, this indeed can happen,
>even without any missing data. Missing data could make the problem worse.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>>project.org] On Behalf Of Willemijn van Eldik
>>Sent: Monday, 26 February, 2018 12:57
>>To: 'r-sig-meta-analysis at r-project.org'
>>Subject: [R-meta] Question about explained variance at 2nd and 3rd level
>>
>>Dear colleagues,
>>
>>Gratefully, I am making use of your R Package Metafor to conduct a
>three-
>>level meta-analysis. I have a question and I hope that you can help me
>>with this question.
>>Shortly, I am conducting several meta-analyses for the association
>>between the interparental relationship and children’s problem behavior.
>>The ES of interest is r (z-transformed) and my dataset concerns 1601 ES
>>from 206 independent samples.
>>I am conducting several univariate moderator analyses, mostly using
>>categorical moderator (dummies).
>>
>>I would like to extract from the moderator analysis how much variance
>one
>>moderator explains at the 2nd and 3rdlevel. My assumption is that I can
>>use the sigma-components, to compare nested models, to calculate this
>>amount of explained variance. Following this reasoning, I would have
>>expected the sigma-component to become smaller in significant moderator
>>models, compared to the overall model (without moderators). However, the
>>sigma-components are not consistently becoming smaller when a
>significant
>>moderator is added to the model. Neither are the sigma-components
>>consistent in amount, e.g. getting larger, is some models including a
>>non-significant moderator.
>>Based on these observations of the output, I am hestitant to use the
>>sigma-components to  evaluate the explained variance by a moderator on
>>each level. My first assumption was that this was caused by missing
>>values in moderator analyses. However, this shows even when no missing
>>values are present. But maybe I missed something or are mistaken
>>somewhere.
>>
>>My questions are;
>>-         -  Is it true that this is caused by missing values and the
>>consequence that these models are not comparable with a different N?
>>-          - Is there another reason (in case there are no missing
>>values)?
>>-          - And most important: is there another way of evaluating the
>>explained variance by a moderator?
>>
>>Thanking you in advance,
>>
>>Best regards,
>>Willemijn van Eldik
>>​
>>PhD Candidate
>>
>>Erasmus University Rotterdam
>>Department of Psychology, Education & Child Studies, ESSB
>>Location Mandeville (T) 16, Burg. Oudlaan 50, Room T16-15
>>Postbus 1738
>>3000 DR  Rotterdam
>>E: vaneldik at essb.eur.nl<mailto:vaneldik at essb.eur.nl> T: 0031104089730