[R-meta] inner-outer-grouping-factor-structure for multiple treatment studies

[Viechtbauer Wolfgang (SP)]

I can only give some general feedback based on the information available.

First of all, the sampling variance of a ROM value is not just a function of the sample size, but also of the (squared) coefficient of variation. That may also differ across studies.

Also, in your model, there are not just weights, but there is an entire weight matrix (see: weights(model, type="matrix")) that depends not only on the V matrix, but also var-cov matrix of the random effects, which is probably also not diagonal here. Therefore, focusing solely on the diagonal of the weight matrix can be a bit misleading.

If there is a lot of heterogeneity in the data, then it can also happen that certain pooled contrasts are not significant even though the individual contrasts are. For RE models, pooling does not necessarily lead to an increase in power.

I am also not sure what kind of random effects you are using, so there could also be problems with how you are specifying the random effects.

The stuff for SMD is not really transferable conceptually to ROM. The total sample size appears in the equation for the variances and covariances for SMD because (in that example) the pooled variance (across all groups within each study) is used to compute the SMD values. But for ROM, the variance (whether pooled or not) is not actually used in computing the measure, so that also does not affect how the variances and covariances are computed.

Best,
Wolfgang

-----Original Message-----
From: Vivien Bonnesoeur [mailto:bonnesoeur.vivien at gmail.com] 
Sent: Tuesday, 05 September, 2017 0:18
To: Viechtbauer Wolfgang (SP)
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] inner-outer-grouping-factor-structure for multiple treatment studies

thanks for your help
I've been computing the variance-covariance as you indicated :

ma.grass <- escalc(m1 = forest_mean, m2 = ref_mean, sd1 = forest_sd, sd2 = ref_sd,
                   n1 = forest_N, n2 = ref_N, measure = "ROM", slab = article)
V <- diag(ma.grass$vi)

and case by case, i filled the covariance with
V[row,column] <- V[column,row] <- forest_sd[row]^2 / (forest_N[row] * forest_mean[row]^2)
or
V[row,column] <- V[column,row] <- ref_sd[row]^2 / (ref_N[row] * ref_mean[row]^2)

I was able to fit a model but I'm not convinced about the result for several reasons :

-- it does not confirm my intuition for the comparison between afforestation and ungrazed grassland which did not show a significant result while most of the contrast where significantly <0

-  the weights of the different contrasts also looks strange. For example Breme's study has only 1 contrast and have a total number of sample = 126, much more than the other studies but its weight is only 2.81% . On the contrary, studies with several contrast can present  individual contrast weights higher than 3% (and the total weight of the study higher than 15%) while the their total number of sample is twice as less as Breme. If I adjust  a fixed-effect model, Breme indeed has a much stronger weight (41%). A fixed-effect model is likely not relevant in my case but the difference of weight with the random effect model is so strong that I have the overall impression that studies with several contrasts are overweighted in the random effect model. 
Actually, in the code you proposed for SMD effect size, the variance-covariance matrix is calculated from the total number of sample from the study (called "Ni" in the example), so that each study with multiple treatment are well weighted. In the code I used for the ROM effect size, there was no such adjustment to account for the real number of samples for studies with several contrasts. 
Is it possible to do such adjustment to get more reasonable weights?

I hope my questions and remarks are not too irrelevant, it's the first time I'm doing a meta-analysis.
best regards
Vivien

On Mon, Sep 4, 2017 at 9:25 AM, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
I am not quite sure why you switched to the standardized mean difference as the outcome measure if your original plan was to use the (log) ratio of means. Switching just because I happen to give some illustrative code for the SMD (instead of ROM) on the metafor website is not really a good reason for such a decision IMHO.

In the end, this is purely a programming issue now. As I mentioned, your dataset is a bit more complex, since in your case, it isn't always the first (or second) group whose data is being reused, but sometimes the first, sometimes the second. So the code that I give on the metafor website isn't going to work for you anyway without some further adjustments.

Again, for ROM, the covariance is simply:

sd^2/(n*mean^2)

from the group whose data is being re-used. escalc() already gives you the variances, which go on the diagonal of the V matrix. So, you just need to find a way to fill in the off-diagonal elements of V using the formula above. You could even get this done with something like:

ma.grass <- escalc(m1 = forest_mean, m2 = ref_mean, sd1 = forest_sd, sd2 = ref_sd,
                   n1 = forest_N, n2 = ref_N, measure = "ROM", slab = article)

V <- diag(ma.grass$vi)

And let's say the first and second row are data from a multi-treatment study with 2 treatment groups and 1 control group, where 'ref' corresponds to the control group (so ref_mean/sd/N[1] and ref_mean/sd/N[2] are actually the same data). Then:

V[1,2] <- V[2,1] <- ref_sd[1]^2 / (ref_N[1] * ref_mean[1]^2)

will fill in the covariance of the first and second row into the V matrix.

Now proceed in the same way for other studies. If the data from the 'forest' group are being reused, then use:

V[row,column] <- V[column,row] <- forest_sd[row]^2 / (forest_N[row] * forest_mean[row]^2)

This isn't elegant and 'hard coded' but it gets the job done.

Best,
Wolfgang