[R-meta] multilevel meta-analysis using metafor

[brauldeq]

I am really sorry for the ongoing confusion on my part. But is there a 
way to use weights corresponding to equation 13 or 17 from Hedges, 
Tipton & Johnson (2010; see 
http://onlinelibrary.wiley.com/doi/10.1002/jrsm.5/abstract) or to assign 
equal weight to correlated within-study effect sizes?
I would like to avoid assigning more total weight to studies with a 
larger number of (dependent) effect sizes than studies with fewer (or 
only one) effect sizes.

Thanks a lot.

Best regards,
Denise


Am 06.09.2017 18:33, schrieb Viechtbauer Wolfgang (SP):
> As I just mentioned in another post, in models with more complex
> random effects structures, there are not just weights, but an entire
> weight matrix. For example, in the model you fitted below, the
> marginal var-cov matrix is block diagonal with vi + sigma^2_sample_nr
> + sigma^2_effect_nr along the diagonal and sigma^2_sample_nr for off
> diagonal elements that correspond to the same level of that factor.
> The weight matrix is then the inverse of that marginal var-cov matrix.
> If the sampling errors had been independent, then this would be the
> 'best' weight matrix (in the sense of giving you the most efficient
> estimates of the fixed effects and giving you appropriate estimates of
> the standard errors) and there would then be no need for any further
> adjustments, manually or statistically.
> 
> In your case, the sampling errors are correlated, but you are fitting
> the model with V=vi, that is, assuming a diagonal V matrix. That will
> lead to less efficient but still unbiased estimates of the fixed
> effects, but invalid estimates of the standard errors, and hence the
> need for using robust(model, cluster=data$sample_nr). But this doesn't
> actually change the weight matrix; it only affects the way the var-cov
> matrix of the fixed effects is computed and hence the computation of
> the standard errors. So, also here there is no need to manually adjust
> any weights.
> 
> And yes, *all else equal*, studies with more effect sizes are weighted
> more than studies with less effect sizes. Let me put this in a
> different way: A study providing more evidence about the phenomenon
> under investigation is getting more weight than a study providing less
> evidence. That seems pretty sensible to me.
> 
> Best,
> Wolfgang
> 
> -----Original Message-----
> From: brauldeq [mailto:brauldeq at hu-berlin.de]
> Sent: Tuesday, 05 September, 2017 13:25
> To: Viechtbauer Wolfgang (SP)
> Cc: r-sig-meta-analysis at r-project.org
> Subject: Re: [R-meta] multilevel meta-analysis using metafor
> 
> I was wondering how the weights are assigned correctly when specifying
> rma.mv(yi, vi, random = ~ 1 | sample_nr/effect_nr, data = data) and
> afterwards using
> robust.rma.mv(model, cluster=data$sample_nr, adjust=T) for a (cluster)
> robust estimation of variances and accurate assessment of standard
> errors.
> 
> Just a little reminder concerning my data structure. I have multiple
> effect sizes (all measured in the same sample) per study. The number of
> effect sizes within one study varies greatly from only 1 up until 30.
> 
> I am asking because not long ago I was quite certain that by specifying
> a two-level random effects model, the dependency of effect sizes within
> studies would automatically be accounted for. Now I know that this is
> not the case and I need to account for the dependency of standard 
> errors
> additionally with the robust() function. However, now I am wondering
> whether the weights are assigned correctly with the approach explained
> above. Or do I need to adjust the weights of studies with multiple
> (dependent) effect sizes manually? Specifically, do studies with more
> effect sizes are weighted more than studies with less effect sizes? 
> This
> would be very problematic in my case.
> 
> Thanks for providing me with more details concerning the weighting of
> effect sizes within two-level random-effect models using the rma.mv()
> function.
> 
> Best regards,
> Denise
> 
> Am 30.08.2017 16:31, schrieb Wolfgang Viechtbauer:
>> Yes, this is indeed how you can approach this.
>> 
>> And yes, if the var-cov structured is misspecified (which it is in
>> your case), then the fixed effects are still estimated unbiasedly
>> (although not as efficiently). The problem is that the SEs of the
>> fixed effects will not be correct. Using robust() allows you to get
>> more appropriate estimates of the SEs (and hence more appropriate
>> tests/CIs).
>> 
>> Best,
>> Wolfgang
>> 
>> On 08/30/2017 03:10 PM, brauldeq wrote:
>> Following
>> https://stackoverflow.com/questions/44811867/multilevel-meta-analysis-using-metafor
>> 
>> I understand that I could at first specify my model using model <-
>> rma.mv(yi, vi, random = ~ 1 | sample_nr/effect_nr, data = data). To
>> solve the problem concerning the covariances of the sampling errors I
>> would hereafter use robust.rma.mv(model, cluster=data$sample_nr, 
>> adjust
>> = T). Would this approach solve my problem?
>> 
>> Am I right to assume that the rma.mv(yi, vi, random = ~1 |
>> sample_nr/effect_nr) function would calculate proper estimate of 
>> effect
>> size but is problematic in terms of sampling errors?
>> 
>> Thanks,
>> Denise
>> 
>> Am 30.08.2017 14:05, schrieb Wolfgang Viechtbauer:
>> Please keep the mailing list in cc.
>> 
>> Yes, this means the subjects overlap, that is, the correlations are
>> computed based on the same sample. In that case, the correlations are
>> correlated. Equations for computing the covariances can be found in:
>> 
>> Steiger, J. H. (1980). Tests for comparing elements of a correlation
>> matrix. Psychological Bulletin, 87(2), 245-251.
>> 
>> There are various cases. Let's say there are four variables: x1, x2,
>> x4, and x4, all measured in the same sample. Then we have the case of
>> non-overlapping variables:
>> 
>> cov(cor(x1,x2), cor(x3,x4))
>> 
>> To compute that covariance, you will need the full 4x4 correlation
>> matrix.
>> 
>> And there is the case of partially overlapping variables, for example:
>> 
>> cov(cor(x1,x2), cor(x1,x3))
>> 
>> To compute the covariance here, you will need cor(x2,x3) (obviously,
>> cor(x1,x2) and cor(x1,x3) you already have, otherwise you would not be
>> interested in their covariance).
>> 
>> Again, the necessary equations can be found in Steiger (1980).
>> 
>> If you do not have the information to compute the covariances, then we
>> are back to the situation where the covariances between the outcomes
>> cannot be computed. See previous posts on how to deal with that. For
>> example:
>> 
>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000097.html
>> 
>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000094.html
>> 
>> 
>> Best,
>> Wolfgang