[R-meta] Violation in non-independece of errors (head to head studies and mutlilevel meta-analysis)?

Mon Mar 5 21:38:59 CET 2018

Update:

I had a closer look at the models and what changes in the estimates:

When I add "id" as another random effect level, I find that sigma^2 is 
0.000 for Outcome and id (or very near to that in the models with 
moderators). Effect estimates for moderators remain nearly unchanged 
(although some CIs change).  To me it doesn't look as if it makes sense 
therefore to include "id" in addition to the original multivariate model.

So, if it is not really wrong, I would prefer to stick to the original 
model.

But why is the effect size estimate different (SMD about 0.1 higher with 
a three-level model or when including the id random effect) and which 
one is correct?

Best,

Emily

Am 05.03.2018 um 17:12 schrieb Emily Finne:
> Dear Wolfgang,
>
>
> may I just chimp into your conversation, since after reading it, I am
> getting quite uncertain about our own analysis...
>
> We have a combination of studies with multiple treatments compared to
> the same control group (in some studies) and of 2 different outcome
> measures (but also only in a subset of the studies, i.e. one outcome was
> present in all studies, the second was additionally present in a subset
> of studies). We first looked at the overall effect and in a next step
> tested different moderators.
>
> We followed a multivariate approach with rma.mv and used the mutivariate
> parametrization as described in the konstantopoulos2011 example on the
> metafor website. So we have:
>
> random = ~ Outcome | study
>
> However,  we also have studies with multiple treatment groups. After
> reading your example code (from you reply below), I am not sure if it
> would be correct to add another random effect for each effect size, i.e.
>
> random = ~ Outcome | study/id following your example below.
>
> We did not do that, because we thought that with Outcome as inner factor
> we have added random variation between the different effect sizes within
> each study (for those cases where more than one effect size is included).
>
> After trying out  random = ~ Outcome | trial/id, however, we get a
> different (higher) overall effect.
>
> And after reading the website example again, I also compared the results
> for a three level (random = ~ 1 | study/Outcome) versus a multilevel
> parametrization (random = ~ Outcome | study).
>
> In fact, these results also differ, and the overall estimated effect
> size for the three-level model is (in terms of robust CIs very nearly)
> the same as for the model with random = ~ Outcome | study/id.
>
> Are we making a mistake if we ignore the additional "id" level random
> effect? Or do we add this random effect mistakenly twice, since we
> already have incorporated random varation within the studies by using
> Outcome as inner factor?
>
> There are, in fact, 2 studies which had both: 2 outcomes but also 2 or 3
> treatment groups. So this may be the part we missed so far by ignoring
> "id" as additional level?
>
> We have 6 studies which had both outcomes but only one treatment group.
> I am therefore not sure if we would overparameterize if we include "id",
> because these trials have two lines in the dataset (2 ids) that also
> stand for the 2 Outcomes.
>
> The variance-covariance-matrix includes covariances for different
> outcomes within the same study, for different treatment groups within
> one study or for both, as appropriate. The profile likelihood plots for
> our orignal multivariate (~ Outcome | study) model looked fine.
>
> Or would it be better to stick to the three level model?  - We describe,
> but not further analyze or discuss, differences between both outcomes,
> because both, though one gives a bit higher estimates, intend to measure
> the same outcome by different instruments. For the analysis of
> moderators which is our main question it makes more sense to look at
> only one moderator effect instead of one for each outcome measure, since
> only some studies used both outcome measures. But of course, we would
> like to take account of the fact that different outcome measures were used.
>
> Would a change in the strategy likely result in changes of the fixed
> effects of moderators?
>
> I hope it is to some extent clear what I mean.  Any help would be very
> much apreciated!
>
> Thanks in advance!
>
>
> Best,
>
> Emily
>
>
>
> Am 05.03.2018 um 10:12 schrieb Viechtbauer Wolfgang (SP):
>> Dear Natan,
>>
>> If you reuse the information from a placebo group to compute multiple effects (i.e., treatment 1 vs placebo, treatment 2 vs placebo, etc.), then this automatically induces dependency in the sampling errors of the estimates. Code to compute the covariance for various effect size measures can be found here:
>>
>> http://www.metafor-project.org/doku.php/analyses:gleser2009
>>
>> So, you need to construct the full V matrix, use rma.mv(), and also include appropriate random effects (at least for studies and for each row of the dataset) in the model. So, something like this:
>>
>> dat$id <- 1:nrow(dat)
>> res <- rma.mv(yi, V, mods = ~ <whatever fixed effects you think are needed>,
>>                 random = ~ 1 | study/id, data=dat)
>>
>> I am a bit confused about:
>>
>>> We are trying to avoid network meta-analysis, given we want our results
>>> to be adjusted by several moderators that affect antidepressant response.
>> Why do you think that network meta-analysis is not compatible with 'adjustment by moderators'?
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>>> project.org] On Behalf Of Natan Gosmann
>>> Sent: Saturday, 03 March, 2018 20:46
>>> To: r-sig-meta-analysis at r-project.org
>>> Subject: [R-meta] Violation in non-independece of errors (head to head
>>> studies and mutlilevel meta-analysis)?
>>>
>>> Hello all,
>>>
>>> We are conducting a large multilevel meta-analysis using the metafor
>>> package considering all RCTs that assessed medications vs placebo for
>>> psychiatric disorders.
>>>
>>> We included all available outcomes from each study and therefore, we are
>>> considering study and assessment instrument (scale) as random variables
>>> in
>>> the model. The yi comes from differences in standardized mean change
>>> between medication and placebo for each study.
>>>
>>> We are trying to avoid network meta-analysis, given we want our results
>>> to
>>> be adjusted by several moderators that affect antidepressant response.
>>>
>>> However, we have doubts about how we can handle head to head studies
>>> (studies with more than one medication) and studies with distinct dosages
>>> of the same medication. We were thinking to just calculate differences
>>> from
>>> placebo (but placebo would be the same group for those studies - would be
>>> the contrast group for more then one medication or dosage group).
>>> Including
>>> study ID as random variable already accounts for violation in
>>> non-independence of errors? Is that an appropriate way of doing that?
>>>
>>> Alternatively, should we select only one medication from head to head
>>> trials?
>>>
>>> I would very much appreciate if you could help us with that.
>>>
>>> Best regards,
>>> Natan
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

-- 
Dr. Emily Finne, Dipl.-Psych.

Universit�t Bielefeld
Fakult�t f�r Gesundheitswissenschaften
AG 4: Pr�vention und Gesundheitsf�rderung
Postfach 10 01 31

D-33501 Bielefeld

Mail: emily.finne at uni-bielefeld.de
http://www.uni-bielefeld.de/gesundhw/ag4

	[[alternative HTML version deleted]]