[R-meta] Three-level meta-analysis of response ratios when there is more than one "control group"

[James Pustejovsky]

Gabriele,

If you are going to use the ROMs approach, Wolfgang's point about using a
block-diagonal covariance matrix is crucial. Accounting for the covariances
between the ROM effect size estimates is necessary for the other parts of
the model (especially the variance components) to make sense.

The Van den Noortgate et al. (2013) paper argues that it is not necessary
to include these sample covariances if you instead include an additional
random effect within the study. The premise is essentially that the
additional random effect makes the model robust to mis-specification of the
covariances. My concern with this argument is that, while it seems to be
true for certain specific models, the robustness property might not hold
generally, and short of doing a simulation study that mimics the conditions
of your meta-analysis, there's not really any easy way to tell whether it
holds. So better to go with the approach that includes the covariances--a
properly specified model--so that the variance component estimates and
standard errors and everything else will come out okay.

However, I think there still might be an issue with the ROM approach as you
have described it. Specifically, if you calculate ROM for each possible
pair of altitudes, then the set of ROM estimates will be perfectly
dependent. Some of the ES estimates will be exact linear combinations of
the other ES estimates. I'm not sure if metafor can even deal with the
resulting variance-covariance matrices, and even if it can it seems like a
tricky and complicated model to interpret. Two possible ways to simplify it
would be: 1) pick an arbitrary "control" condition and compare all of the
other conditions to it (e.g., always treat the highest altitude as the
control, and all of the lower altitudes as treatment conditions) or 2) use
raw means (or log-means) as effect sizes, as I suggested earlier, and which
has the further benefit that the ES estimates are independent (so the van
den Noortgate 3-level meta-analysis model would actually be perfectly
well-specified here).

James

On Wed, Apr 11, 2018 at 4:20 PM, Gabriele Midolo <gabriele.midolo at gmail.com>
wrote:

> Hi James (all),
>
> Thanks again for your exhaustive answer.
> Sorry if I insist, but I still did not understand why multi-level
> modelling via rma.mv is not a good option to deal with non-independence
> in my case (or maybe I missed something from your email?). I had a read to
> Lajeunesse (2011) and Lajeunesse (2016) [here, he describes a practical
> example in R on to build variance covariance matrix when you have multiple
> treatments and one control and use them with metafor]. However, I saw e.g.
> in Van den Noortgate et al (2013) that such issue can be dealt with
> multiple-level meta-analytic approaches too (?), or at least that is my
> interpretation.
>
> So, based on Lajeunesse's example, I have simulated data that could
> potentially reflect the data of my meta-analysis (I took the liberty to
> attach these, hope it is not an issue with the rules of r-sig-meta-analysis
> mailing list).
> Here I have three studies that report treatment and control change (X_T
> and X_C, respectively). The yi and vi were calculated with the "ROM" option
> in escalc().
> If I had a case where yi of each study shares the same (only one) control,
> I would fit the model with following:
>
> "res1<-rma.mv(yi,vi,data=dat,random=~ 1 | Study/effect_ID)"
>
> However, I should account for the fact that I have multiple control group
> for each yi in each study, so wouldn't be correct to add a third level
> "commonControl_ID"? i.e.:
>
> "res2<-rma.mv(yi,vi,data=dat,random=~ 1 | Study/commonControl_ID/effect_
> ID)"
>
> Which should be identical to:
>
> "res3<-rma.mv(yi,vi,data=dat,random=list(~ 1 | effect_ID, ~ 1 |
> commonControl_ID, ~ 1 | Study))"
>
> (... at least, the model output based on my data example looks like they
> are). Or maybe all of what I have put above does not make sense and I am
> misunderstanding the usage of multi-level modeling?
>
> Thanks for your patience,
> Gabriele
>
> On 10 April 2018 at 18:38, James Pustejovsky <jepusto at gmail.com> wrote:
>
>> Gabriele,
>>
>> I see what you mean. Your explanation suggests that there would be high
>> variance in the outcome from study to study, due to examining different
>> species or other aspects of the study's design. I don't think that this
>> precludes using the raw means as effect sizes---it just means that there
>> would be a large between-study variance component. I think this approach
>> would still make it easier to model how the ES might depend on covariates
>> that have variation within a given study (such as altitude level).
>>
>> The difficulty with using the ROM effect size is that when you compare
>> multiple conditions to a single control condition, there is correlation in
>> the effect size estimates. In the notation of your original email, you
>> would have
>> cov(yi1, yi2) != 0
>> cov(yi1, yi3) != 0
>> cov(yi2, yi3) != 0
>> In fact, there would be very strong dependence because yi3 = yi2 - yi1.
>> So if you use this approach, it would be critical to account for the
>> sampling covariance between the effect sizes. Lajeunesse (2011) gives
>> formulas for the covariances that you'd need.
>>
>> James
>>
>> Lajeunesse, M. J. (2011). On the meta-analysis of resposne ratios for
>> studies with correlated and multi-design groups. Ecology, 92(11), 2049–2055.
>>
>> On Tue, Apr 10, 2018 at 10:22 AM, Gabriele Midolo <
>> gabriele.midolo at gmail.com> wrote:
>>
>>> Dear James (all),
>>>
>>> Thanks for your ideas, very appreciated.
>>> I understand what you mean, but SLA can be calculated on different
>>> species which are likely to report very different magnitude depending on
>>> the species. Plus I think the ratios calculated with "ROM" should provide a
>>> standardized measure of how much a single species' SLA changes in response
>>> to altitudinal shift, that is what I think might be interesting to measure
>>> in the context of my study.
>>> I am not unkeen to use your option of course, I am just wondering if a
>>> three-level meta-analysis could deal with a type of independency described
>>> above, or it can work with data that share multiple treatment "doses"
>>> compated to a single control...
>>>
>>> Gabriele
>>>
>>> On 10 April 2018 at 16:21, James Pustejovsky <jepusto at gmail.com> wrote:
>>>
>>>> Gabriele,
>>>>
>>>> If all of the studies use SLA as the outcome measure, would it make
>>>> sense to model the mean levels directly (instead of calculating
>>>> proportionate changes)? In other words, use the mean SLA level as the
>>>> effect size. If it makes sense conceptually, I think this would be a good
>>>> way to go because then the effect size estimates at each altitude level
>>>> would be independent of each other, conditional on the true mean levels for
>>>> that study, and a three-level meta-analysis model would be a natural fit.
>>>>
>>>> James
>>>>
>>>> On Sun, Apr 8, 2018 at 11:12 AM, Gabriele Midolo <
>>>> gabriele.midolo at gmail.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I have a question that is more methodological but somehow related to
>>>>> metafor.
>>>>> I want to conduct an (ecological) meta-analysis on specific leaf area
>>>>> (SLA) response to increased altitdue (i.e. elevation) in mountain
>>>>> ecosystems. Primary studies selected report the mean (+ SE and sample size)
>>>>> of SLA sampled at different altitudinal levels. The picture attached is an
>>>>> example of how row primary data are normally reported in the articles
>>>>> (modified, from Seguí et al 2018, fig.1c [
>>>>> https://doi.org/10.1007/s00035-017-0195-9]).
>>>>> The A, B and C (in red) values represents the mean values of SLA
>>>>> calculated at 1900, 2200 and 2350 m above the sea level (i.e. altitude)
>>>>> that should, in my opinion, be suitable for calculating log-transformed
>>>>> response ratios (RR) indicating how much SLA increases/decreases compared
>>>>> to a population of plants sampled to a lower altitiude. Thus, given the
>>>>> design of such studies, I propose that multiple RR (yi) must be calulcated
>>>>> within each study as follows:
>>>>>
>>>>> yi1= ln(B/A)
>>>>> yi2=ln(C/A)
>>>>> yi3=ln(C/B)
>>>>> ...
>>>>> if a D value would have been reported by the authors, sampled to a
>>>>> higher altitdue than 2350 m, then I woul also calculate yi4=ln(D/A),
>>>>> yi5=ln(D/B), yi6=ln(D/C) for this study.
>>>>>
>>>>> This approach make sense to me because there is no "proper" control
>>>>> and treatment and you are not just interested to estimate SLA changes by
>>>>> comparing mean values reported at higher altitudes with only the one
>>>>> sampled at the lowest altitudinal level (yi1,yi2), but also between higer
>>>>> altitudinal levels (yi3). This is also supposed to allow to look in
>>>>> meta-regession how the altitudinal shift (so, the difference in altitudes
>>>>> e.g. 300m for yi1) affect the effect size responses. So - and here finally
>>>>> comes my question - with rma.mv I should be able to safely account
>>>>> for non-independence by fitting a model with the "random =~1|Experiment/ID"
>>>>> structure (?). Is this type of data suitable for three-level mixed-effect
>>>>> meta-analysis? I used already this structure in a previous meta-analysis I
>>>>> conducted in the past, but back then I was working with multiple treatments
>>>>> compared to just one single control in each study.
>>>>> I see some similar meta-analysis in the past have used the r-to-z
>>>>> transformed effect size and focused on the correlation - in my case -
>>>>> between altitude and SLA, but not sure this is what I would like to
>>>>> investigate in the first place...
>>>>>
>>>>> Hope I was clear, and my apologies if I was messy.
>>>>>
>>>>> Thanks a lot for reading this
>>>>> Gabriele
>>>>>
>>>>>
>>>>> ​
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-meta-analysis mailing list
>>>>> R-sig-meta-analysis at r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>
>>>>>
>>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20180412/1b5d1719/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: =?UTF-8?B?U2VndcOtX2V0YWxfMjAxOF9leGFtcGxlLlBORw==?=
Type: image/png
Size: 35051 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20180412/1b5d1719/attachment-0001.png>