[R-meta] Calculating the covariance structure for multilevel meta-analysis using the raw mean

[Akifumi Yanagisawa]

Thank you for answering my question, Wolfgang. 
I see. Now I understand that it does not make sense to just specify diagonal covariance structure without putting the V matrix for this case. 

Thank you very much for your time and support. I feel more hopeful to conduct my analysis now. I will learn cluster-robust inference method. 

I hope you have a great day today.
Best regards,
Aki


> On Feb 26, 2018, at 11:58 AM, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
> 
> I am not sure I understand your question. The covariance I was talking about goes into the V matrix. What you specify for 'struct' is something totally different. Regardless of what you put into V, it makes very little sense to set struct = "DIAG", since that assumes that true effects for the same study are independent.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: Akifumi Yanagisawa [mailto:ayanagis at uwo.ca]
>> Sent: Monday, 26 February, 2018 16:34
>> To: Viechtbauer Wolfgang (SP)
>> Cc: r-sig-meta-analysis at r-project.org
>> Subject: Re: Calculating the covariance structure for multilevel meta-
>> analysis using the raw mean
>> 
>> Dear Wolfgang,
>> 
>> Thank you for your quick response and providing very straightforward
>> answers. I will read the past post on the mailing list to learn how to
>> use cluster-robust inference methods.
>> 
>> Just a curiosity, I initially thought I could just fit multilevel models
>> specifying covariance structure as diagonal, instead of specifying V (Or,
>> the exact covariance structure).
>> 
>>>>>> res <- rma.mv (yi, vi, mods = ~ treatment - 1, random = ~treatment |
>> study, struct = "DIAG", data = dat)
>> 
>> What do you think would be the problem with this approach? Do you think
>> it could be acceptable to some extent?
>> 
>> Thank you,
>> Aki
>> 
>>> On Feb 26, 2018, at 10:03 AM, Viechtbauer Wolfgang (SP)
>> <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
>>> 
>>> Thanks for clarifying. For a within-subjects design, you would need the
>> correlation between the measurements under the two treatments in order to
>> compute the covariance of the means. The same applies to the case where a
>> group is measured twice.
>>> 
>>> The problem is of course that these correlations are not reported.
>> Strategies for dealing with this problem have been covered extensively in
>> the past on this mailing list. The most straightforward approach would be
>> to ignore the covariances during the model fitting and then use cluster-
>> robust inference methods.
>>> 
>>> Best,
>>> Wolfgang
>>> 
>>>> -----Original Message-----
>>>> From: Akifumi Yanagisawa [mailto:ayanagis at uwo.ca]
>>>> Sent: Monday, 26 February, 2018 15:52
>>>> To: Viechtbauer Wolfgang (SP)
>>>> Cc: r-sig-meta-analysis at r-project.org
>>>> Subject: Re: Calculating the covariance structure for multilevel meta-
>>>> analysis using the raw mean
>>>> 
>>>> Dear Wolfgang,
>>>> 
>>>> Thank you for replying to my question!
>>>> 
>>>> To your first question, yes for some studies and no for the other
>>>> studies. Some studies used within-participant design, where the same
>>>> group of people engaged in several different treatments (i.e., word
>>>> learning activities). The other studies used between-participant
>> design,
>>>> where one group of people only experienced one treatment.
>>>> 
>>>> For your second question, yes to your assumption, the same group of
>>>> people was involved in both ‘measurement1’ and ‘measurement2’, and the
>>>> group sizes are the same.
>>>> But, not to your question. If studies used both ‘measurement1’ and
>>>> ‘measurement2’, all of the participants’ learning gains were tested
>> with
>>>> both measurements. So the 28 people involved in 'measurement1’ are the
>>>> SAME group of people as the 30 (OR, 28) people involved in
>>>> ‘measurement2’.
>>>> (Sorry for the confusion. The number of participants should have been
>> the
>>>> same; let’s say 28 for here.)
>>>> 
>>>> I am glad to hear that I can regard the learning gains from studies
>> that
>>>> used between-participant design as independent observations and
>> include
>>>> them for analysis. Do you think it would be possible for me to include
>>>> studies that used within-participant design as well?
>>>> 
>>>> Thank you very much for your time and support.
>>>> Best regards,
>>>> Aki
>>>> 
>>>>> On Feb 26, 2018, at 9:17 AM, Viechtbauer Wolfgang (SP)
>>>> <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
>>>>> 
>>>>> Dear Aki,
>>>>> 
>>>>> For the first case, are the 28 people in row 1 a different group of
>>>> people than the 30 people in row 2? Then the sampling errors for the
>> two
>>>> means are independent and you do not have to worry about the
>> covariance.
>>>>> 
>>>>> For the second case (with multiple measurements), I would have
>> assumed
>>>> that the same group of people was involved in 'measurement1' and
>>>> 'measurement2' but (assuming no missing data) then the group sizes
>> should
>>>> be the same, which they are not in your illustrative data. So, then I
>>>> would have the same question: Are the 28 people involved in
>>>> 'measurement1' a different group of people than the 30 people involved
>> in
>>>> 'measurement2'?
>>>>> 
>>>>> Best,
>>>>> Wolfgang
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>>>>>> project.org] On Behalf Of Akifumi Yanagisawa
>>>>>> Sent: Monday, 19 February, 2018 14:57
>>>>>> To: r-sig-meta-analysis at r-project.org
>>>>>> Subject: [R-meta] Calculating the covariance structure for
>> multilevel
>>>>>> meta-analysis using the raw mean
>>>>>> 
>>>>>> Dear all,
>>>>>> 
>>>>>> I am currently conducting a multilevel meta-analysis with the
>> metafor
>>>>>> package using the raw mean.
>>>>>> The following is sample data which describes the characteristics of
>> my
>>>>>> data. Please note the actual data set includes more studies.
>>>>>> 
>>>>>> ## data structure ##
>>>>>> dat <- data.frame(
>>>>>> id = c(1,2,3,4,5, 6, 7),
>>>>>> study = c("study1","study1", "study2", "study2", "study3", "study3",
>>>>>> "study4"),
>>>>>> treatment = c("treatment1","treatment1", "treatment2", "treatment3",
>>>>>> "treatment3", "treatment4", "treatment4"),
>>>>>> mi = c(5.3, 5.1, 2.2, 3.4, 2.5, 5.1, 3.3),
>>>>>> sdi = c(1.2, 1.3, 0.5, 3.1, 0.7, 2.1, 0.8),
>>>>>> ni = c(28, 30, 20, 20, 42, 43, 120))
>>>>>> require(metafor)
>>>>>> dat <- escalc(data = dat, measure = "MN", mi = mi, sdi = sdi, ni =
>> ni)
>>>>>> dat
>>>>>> 
>>>>>> As you can see in the above data, some studies examined only one
>> type
>>>> of
>>>>>> treatment. I am trying to specify the model using the the following
>>>> code.
>>>>>> 
>>>>>> res <- rma.mv (yi, vi, mods = ~ treatment - 1, random = ~treatment |
>>>>>> study, struct = "DIAG", data = dat)
>>>>>> 
>>>>>> However, I cannot figure out how to calculate the covariance
>> structure
>>>>>> for this case, i.e., some studies include only one treatment. I am
>>>>>> thinking that I probably cannot use the same approach as network
>> meta-
>>>>>> analysis. I would appreciate it if I could learn how to calculate
>> the
>>>>>> covariance matrix for this case.
>>>>>> 
>>>>>> Also, some of the studies measured the outcome using two types of
>>>>>> measurements. None of these studies reported a correlation between
>>>>>> measurement 1 and measurement 2. Would it be possible for me to
>>>> include
>>>>>> this type of measurement as a factor while somehow specifying the
>>>>>> covariance matrix for this as well? Or, should I just analyze the
>> data
>>>>>> separately for each type of measurement?
>>>>>> 
>>>>>> ## data structure 2 ##
>>>>>> dat <- data.frame(
>>>>>> id = c(1,2,3,4,5, 6, 7),
>>>>>> study = c("study1","study1", "study2", "study2", "study3", "study3",
>>>>>> "study4"),
>>>>>> treatment = c("treatment1","treatment1", "treatment2", "treatment3",
>>>>>> "treatment3", "treatment4", "treatment4"),
>>>>>> measurement = c("measurement1","measurement2", "measurement1",
>>>>>> "measurement1", "measurement2", "measurement2", "measurement1"),
>>>>>> mi = c(5.3, 5.1, 2.2, 3.4, 2.5, 5.1, 3.3),
>>>>>> sdi = c(1.2, 1.3, 0.5, 3.1, 0.7, 2.1, 0.8),
>>>>>> ni = c(28, 30, 20, 20, 42, 43, 120))
>>>>>> dat <- escalc(data = dat, measure = "MN", mi = mi, sdi = sdi, ni =
>> ni)
>>>>>> res <- rma.mv(data = dat ,yi, vi, mods = ~ treatment + measurement -
>> 1,
>>>>>> random = ~treatment|study, struct = "DIAG”)
>>>>>> 
>>>>>> Thank you very much for your help,
>>>>>> Aki