[R-meta] complex data structure for a meta-analysis

[Viechtbauer Wolfgang (SP)]

Variance components might be estimated to be equal to zero. This does not indicate any problems with model fitting or that you did anything wrong.

Best,
Wolfgang

-----Original Message-----
From: Yogev Kivity [mailto:yogev_k at yahoo.com] 
Sent: Wednesday, 08 November, 2017 6:19
To: r-sig-meta-analysis at r-project.org; Viechtbauer Wolfgang (SP)
Subject: Re: [R-meta] complex data structure for a meta-analysis

Thanks a lot. this is very helpful!

I was able to create the V matrix, but I did run into problems fitting the model. I used the following code:

V <- impute_covariance_matrix(vi = DataF$vi, cluster = DataF$GroupID, r = 0.5, return_list = F)

rma.mv(yi, V, random = ~ 1 | Study/GroupID/EffectSizeID, data=DataF)

If I use the raw variances (e.g., ignoring the correlation between residual variances), the model runs correctly. However, If I use the impute_covariance_matrix command then the variance at the group level (sigma^2.2) becomes zero for whatever reason.
Note that GroupID is a unique identifier (i.e., it does not repeat across groups).

I was wondering whether someone would have a clue of what I might be doing wrong. I can also post some sample data if needed.

Thanks,
Yogev

--
Yogev Kivity, Ph.D. 
Postdoctoral Fellow 
Department of Psychology 
The Pennsylvania State University 
Bruce V. Moore Building 
University Park, PA 16802 
Office Phone: (814) 867-2330

On Thursday, October 26, 2017, 4:46:22 PM EDT, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote: 

See: https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000094.html

Best,
Wolfgang

-----Original Message-----
From: Yogev Kivity [mailto:yogev_k at yahoo.com] 
Sent: Thursday, 26 October, 2017 22:38
To: r-sig-meta-analysis at r-project.org; Viechtbauer Wolfgang (SP)
Subject: Re: [R-meta] complex data structure for a meta-analysis

Yes, there is no overlap.

So, last question (I think), say that I want to use a specific value for the correlation between sampling errors (say, r = .5) and the sampling errors of the ES are known, is there a way to compute the covariance when I build the V matrix using the bldiag function?

thanks again,
Yogev

--
Yogev Kivity, Ph.D. 
Postdoctoral Fellow 
Department of Psychology 
The Pennsylvania State University 
Bruce V. Moore Building 
University Park, PA 16802 
Office Phone: (814) 867-2330

On Thursday, October 26, 2017, 9:21:53 AM EDT, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote: 

Yes, as long as the different groups consist of different subjects (no overlap at all), then the sampling errors can be assumed to be independent. So, for example, if, for a given study, we have an effect size estimate for the subgroup of males and an effect size estimate for the subgroup of females, then there is no overlap and the sampling errors are independent.

On the other hand, when the same group is measured with respect to multiple outcomes or at multiple time points, then there is obviously overlap (in fact, it's the exact same group). Another case of overlap occurs when you have a group of subjects subdivided into, let's say, 3 groups and one computes two effect sizes (e.g., standardized mean differences, log odds/risk ratios, etc.) that contrasts group 1 with group 3 and group 2 with group 3 (i.e., the 'multiple treatment group case'). In all those cases, sampling errors are no longer independent.

Best,
Wolfgang

-----Original Message-----
From: Yogev Kivity [mailto:yogev_k at yahoo.com] 
Sent: Thursday, 26 October, 2017 14:54
To: r-sig-meta-analysis at r-project.org; Viechtbauer Wolfgang (SP)
Subject: Re: [R-meta] complex data structure for a meta-analysis

Hi Wolfgang!

Thanks for the speedy reply!

I want to elaborate more on 1a. I think what I'm struggling to understand is how our design of 4-level model affects the way I should construct the V matrix. Again, we have multivariate effects that are nested within group, which in turn, are nested within studies. Can I assume that effect sizes of different groups within the same study are independent as far as their sampling errors are concerned?

Best,
Yogev

--
Yogev Kivity, Ph.D. 
Postdoctoral Fellow 
Department of Psychology 
The Pennsylvania State University 
Bruce V. Moore Building 
University Park, PA 16802 
Office Phone: (814) 867-2330

On Thursday, October 26, 2017, 4:35:45 AM EDT, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote: 

1) Multivariate data have in essence automatically a multilevel structure. When I use the term 'multivariate', I use it to denote a dataset where multiple estimates are computed for the same sample (e.g., due to multiple dependent variables, multiple time points, multiple treatment groups compared against a single control group) and hence the sampling errors are no longer independent. Besides the dependency in the sampling errors, the data have a multilevel structure, since the estimates are nested within studies.

a) In multivariate data, there are two sources of dependency: The dependency in the sampling errors (which should be incorporated into the V matrix) and potential dependency in the underlying true effects -- the latter can be accounted for by using appropriate random effects. So, in your case, it seems like you would want random intercepts for study, group, and each individual estimate.

b) Yes, that is fine. Berkey et al. (1998) is a special case where we have both outcomes for each study. But that's not necessary (and in fact rather unusual).

If there are a lot of different outcome types, you may not be able to use a fully unstructured var-cov matrix for the random effects (i.e., struct="UN").

2) You can always add up or take the average of variance components if you want to compute some kind of overall I^2-type measure. I^2 is just a proportion (out of some measure of total variability), so the principle is easily extended to pretty much any model.

Best,
Wolfgang

-----Original Message-----
From: Yogev Kivity [mailto:yogev_k at yahoo.com] 
Sent: Wednesday, 25 October, 2017 21:56
To: r-sig-meta-analysis at r-project.org; Viechtbauer Wolfgang (SP)
Subject: Re: RE: [R-meta] complex data structure for a meta-analysis

Dear Wolfgang and all,

Thanks a lot for your detailed reply - this is very helpful! I have read the posts in the links and I feel like I have a better sense of the issues I am dealing with now. I do have several outstanding issues that I am hoping to get peoples’ opinion on.

1. Yes, my data are indeed multivariate, but I think it is also multilevel, because most studies had multiple groups (i.e. treatment arms) nested within the study. We are interested in examining the correlation between attachment style and treatment outcome within each group, and then as a second step to examine whether the effect size differs by treatment type (group level moderator). In addition, there is substantial variability across studies in the number of outcome measures (e.g., some report only one while others report 4-5) as well as the construct being measured (e.g., some measure depression and anxiety, while others measure functioning and quality of life). These issues raise several questions:

a. is there a way to account for the dependency between groups (in addition to simply including group as a level in the model)?

b. Say I choose to specify a ‘guestimated’ var-cov matrix to account for the dependencies among sampling variances, is it ok to have a different number of outcome measures, and different types of outcome measures in each study? The Berkey et al., 1998 study, for example, had the same number and type of outcome measures for all studies.

2. regarding I^2 – again, in my case (see above) I have both multilevel and multivariate data so I am not sure if the approaches mentioned in the link may be appropriate for my data. In general, I am interested in a general I^2 across outcome measures, and not necessarily by measure.

Thanks in advance,
Yogev

--
Yogev Kivity, Ph.D. 
Postdoctoral Fellow 
Department of Psychology 
The Pennsylvania State University 
Bruce V. Moore Building 
University Park, PA 16802 
Office Phone: (814) 867-2330

On Saturday, September 23, 2017, 5:51:56 PM EDT, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote: 

1) You do have what I would call 'multivariate' data -- that is, multiple outcomes for the same sample. Since you say that you do not have the information needed in order to compute the covariances between multiple outcomes for the same sample, a possible strategy to account for the dependency is the use cluster robust inferences. This has been discussed at quite some length in previous posts on this mailing list, so I would encourage you to look through the archives.

2) Similarly, this has come up before. There is no general concensus on how I^2 can be extended to more complex models. I have written down some ideas here:

http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate

I cannot tell you whether this is the best/right way.

3) A standard funnel plot isn't wrong, it just does not give any indication of what points are independent vs dependent. I am not aware of any suggestions on how a funnel plot could be drawn that does provide such an indication (maybe connecting dependent estimates with lines?).

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Yogev Kivity
Sent: Wednesday, 20 September, 2017 13:42
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] complex data structure for a meta-analysis

Hi everyone,I am running a meta-analysis using 'metafor' and I came across several questions that I could not find answers
for in 'metafor's documentation.

In short, were are examining psychotherapy data, and how a specific measure collected at the beginning of treatment (attachment style in relationships) predicts outcome of therapy as measured at post-treatment. Both measures are usually dimensional, so we are using Pearson's r which we then convert to Fisher's z.

The design of the meta-analysis is multilevel and multivariate in that each study usually includes several different treatment groups with different patients, as well as several subscales of attachment (e.g., level of anxiety in attachment and level of avoidance in attachment) and several measures of outcome at post-treatment (e.g., anxiety, depression etc.). This is complicated by the fact that studies rarely use the same attachment and outcome measures, and for the most part, we do not have data on the covariance among these measures.

I am assuming that our design is most similar to Konstantopoulos (2011), but we have an additional level of effect sizes repeated within groups, so basically we have multiple effect size per treatment arm, nested within treatment arm, which in are turn nested within study. Would that be correct?

My main questions are:
1. what would be the best approach for modeling all of these levels of analyses, while taking into account the fact that the effect sizes within treatment arm are likely no independent. My understanding is that usually multivariate is interpreted to mean multiple outcome measures, but in our case we have multiple outcome as well as multiple predictors.

2. How should I squared be calculated for such models?

3. is there an extension of funnel plots to multi-level models that could reliably represent the data? I guess that using the standard funnel plot ignores the mutilevel structure of the data, is that correct?

Best,
Yogev