[R-meta] "impute-the-correlation + robustness + sensitive analysis" strategy

[Michael Dewey]

Dear Celia

One thing which you could do while you are waiting is to do the 
imputations assuming all the correlations are equal. If you do this for 
a range of plausible values it may be that the substantive result of 
your meta-analysis is unaffected and so you can be confident in your 
interpretation. Most techniques are much more sensitive to some inputs 
than to others.

Michael

On 23/01/2018 20:35, Célia Sofia Moreira wrote:
> Dear James,
> 
> Thank you very much for your quick reply.
> 
> You write that "assume r = 0.7 for the inter-correlation between T1 through
> T4, but then assume r = 0.2 for the correlation between these measures and
> T5." However, I can not assume two different correlations simultaneously
> with the impute_the_covariance function. So, I presume you are suggesting
> to run the "impute_the_covariance + rma.mv" successively for r=.2,..., r=.7
> (steps of .1 or .2), isn't is? In affirmative case, is the R code I wrote
> - in my previous message - the adequate?
> 
> About creating the variance-covariance matrix by hand is a very complex
> procedure because correlations across domains vary from paper to paper, and
> I would have to consider a reasonable number of matrices resulting from
> different correlations combinations.
> 
> I would be grateful if you could provide me a generalization of the
> impute_the_covariance function to my use-case. I will definitely wait.
> Thank you! Please let me know when it's ready :)
> 
> Kind regards,
>   celia
> 
> 2018-01-23 15:44 GMT+00:00 James Pustejovsky <jepusto at gmail.com>:
> 
>> Celia,
>>
>> If you have reasonable prior information about the correlation between
>> certain tests, which suggests that some tests are more highly correlated
>> then others, then I would definitely recommend using that information.
>>
>> In principle, you can still use the "impute-the-correlation" strategy even
>> while assuming unequal correlations between certain tests. For instance, as
>> you suggested, you might assume r = 0.7 for the inter-correlation between
>> T1 through T4, but then assume r = 0.2 for the correlation between these
>> measures and T5. For sensitivity analysis, you could vary these
>> correlations by adding/subtracting 0.1 or 0.2 from each (off-diagonal)
>> cell.
>>
>> Another useful sensitivity analysis would be to assume zero correlations
>> between tests from different domains and then also use the struct = "DIAG"
>> argument in rma.mv. This amounts to estimating separate (marginal) models
>> for each domain. If you get very different average effect estimates than
>> you do with the full multivariate model, then it would indicate that the
>> results are going to be fairly sensitive to the assumption you make about
>> the cross-domain correlations.
>>
>> Of course, implementing these approaches takes a bit more work. One
>> approach would be to create the variance-covariance matrices for each study
>> "by hand," and then store them in a list that can be fed into rma.mv.
>> This is tedious but might it be the easiest way to go. I have an idea for
>> how to make the impute_covariance_matrix() function more helpful for your
>> use-case, but it will be a week or two before I can get to it.
>>
>> James
>>
>>
>>
>> On Tue, Jan 23, 2018 at 6:16 AM, Célia Sofia Moreira <
>> celiasofiamoreira at gmail.com> wrote:
>>
>>> Dear all,
>>>
>>> I am performing a meta-analysis for the first time, and I would be
>>> grateful
>>> if you could give me some recommendations / suggestions. I apologize for
>>> this long description:
>>>
>>> I have 20 different papers and, because some papers have more than one
>>> experiment, 23 experiments are considered, which correspond to 23
>>> different
>>> samples. The complexity of my case is that each experiment includes
>>> different domains, and each domain includes several tests. These domains
>>> are latent variables in the individual studies (tests are the indicators).
>>> I'm interested in studying the effects in five different domains. Thus,
>>> for
>>> each domain, I usually collected more than one test type measure.
>>>
>>> Finally, each sample includes an experimental and a control group. The
>>> effects were estimated using Hedges' g (adjusted) and then calculating the
>>> difference y between groups. In these calculations, I followed this
>>> valuable example:
>>> http://www.metafor-project.org/doku.php/analyses:morris2008
>>>
>>> Unfortunately, papers do not report correlations/covariances between test
>>> measures and, so, for each y, I only have the estimated variance v.
>>>
>>> So, my database looks like:
>>>
>>> ID ----- Paper --------- Sample ------- Domain ------- Test ------- y ----
>>> v  ----  etc.
>>>
>>> 01 ------- P1  ----------  S1 ----------- D1 ---------- T1 -------- ...
>>> 02 ------- P1  ----------  S1 ----------- D1 ---------- T2 -------- ...
>>> 03 ------- P1  ----------  S1 ----------- D2 ---------- T3 -------- ...
>>> 04 ------- P1  ----------  S1 ----------- D4 ---------- T4 -------- ...
>>> 05 ------- P1  ----------  S1 ----------- D4 ---------- T5 -------- ...
>>> 06 ------- P1  ----------  S1 ----------- D4 ---------- T6 -------- ...
>>>
>>> 07 ------- P2  ----------  S2 ----------- D2 ---------- T3 -------- ...
>>> 08 ------- P2  ----------  S2 ----------- D2 ---------- T7 -------- ...
>>> 09 ------- P2  ----------  S2 ----------- D2 ---------- T8 -------- ...
>>> 10 ------- P2  ----------  S2 ----------- D3 ---------- T9 -------- ...
>>> 11 ------- P2  ----------  S2 ----------- D4 ---------- T4 -------- ...
>>> 12 ------- P2  ----------  S2 ----------- D4 ---------- T5 -------- ...
>>> 13 ------- P2  ----------  S2 ----------- D5 ---------- T0 -------- ...
>>>
>>> 14 ------- P3  ----------  S3 ----------- D1 ---------- T1 -------- ...
>>> 15 ------- P3  ----------  S4 ----------- D1 ---------- T1 -------- ...
>>> 16 ------- P3  ----------  S4 ----------- D2 ---------- T3 -------- ...
>>>
>>> .....
>>>
>>> I would like to assess the significance of these five domains' effects as
>>> well as their correlation, using a meta-analyst approach. D1 is the
>>> central
>>> domain. Thus, I thought about performing a multivariate multilevel random
>>> meta-analysis model and then using the correlation matrix between true
>>> effects provided in the output. Since I have no sampling covariances, I
>>> used the "impute-the-correlation" strategy, by James Pustejovsky, together
>>> with robust and sensitive analysis. Because I'm interested in the domains
>>> (latent variables), the R code I'm using is the following:
>>>
>>> Vlist <- impute_covariance_matrix(vi = dat$v, cluster = dat$Sample, r =
>>> 0.2)   # Followed by r=.4, r=.6 and r=.8
>>> meta1 <- rma.mv(y ~ 0 + Domain, V=Vlist, random = ~ Domain | Sample,
>>> struct
>>> = "UN", data = dat)
>>> meta1r <- robust(meta1, cluster = dat$Sample); summary(meta1r);
>>> r1<-cov2cor(meta1r$vb); R1 <- round(c1,2)
>>>
>>> Does it make sense to you? Do you recommend corrections?
>>>
>>> Introducing 2 moderators in the model, I got interesting results with this
>>> "impute-the-correlation"+robust+sensitive strategy (ranging r from .2 to
>>> .8), namely, the significance of D1 true effect.
>>>
>>> However, the "impute-the-correlation" assumes equal correlations across
>>> studies and across outcomes. I've been searching in literature, and I
>>> found
>>> some papers reporting a high correlation between D2, D3, D4, D5: r=-6-.7
>>> between pairs, and r=.7-.8 intra-pairs. Their correlation with D1 is
>>> lower,
>>> and even lower for one of the pairs.
>>>
>>> Given this different range of correlations among the outcomes, do you
>>> think
>>> that the results I got with this strategy are truthful? Or do you think it
>>> is better to perform the analysis in different steps, for different
>>> subsets
>>> of variables in the multivariate multilevel model, for example,
>>> considering
>>> two at a time, or D1 with each pair separately?
>>>
>>> Thank you very much for your attention!
>>>
>>>          [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>>
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael
http://www.dewey.myzen.co.uk/home.html