[R-meta] Covariance-variance matrix when studies share multiple treatment x control comparison

Ju Lee juhyung2 @end|ng |rom @t@n|ord@edu
Fri Sep 27 16:46:06 CEST 2019

```Dear James, Wolfgang,

Thank you very much for this information.
I have one question extending from this is: While I run my main mixed modes always using var-covar. matrix (to account for shared study groups within each study),
it is acceptable that my egger-like regression does not incorporate this structure, but rather just use sqrt(1 / n1 + 1 / n2)  as precision (instead of sqrt(diag(v.c.matrix)) like Wolfgang suggested as one possibility) and use p-value for precision term (precision.2 which is p=0.2382) to determine the asymmetry?

prec.<-function(CN,TN){
pr<-sqrt((1 / CN) + (1/TN))
return(pr)
}
precision.2<-prec.(MHF\$n.t, MHF\$n.c)
> egger.pr2.frag<-rma.mv(residuals~precision.2,var,data=MHF,random =list( ~ 1 | Study, ~1|Id),  subset=(Spatial.scale.2=="Fragmentation scale"))
> egger.pr2.frag

Multivariate Meta-Analysis Model (k = 285; method: REML)
Variance Components:
estim    sqrt  nlvls  fixed  factor
sigma^2.1  0.4255  0.6523     73     no   Study
sigma^2.2  0.3130  0.5595    285     no      Id
Test for Residual Heterogeneity:
QE(df = 283) = 1041.1007, p-val < .0001
Test of Moderators (coefficient(s) 2):
QM(df = 1) = 1.3909, p-val = 0.2382

Model Results:
estimate      se     zval    pval    ci.lb   ci.ub
intrcpt        0.0529  0.1617   0.3274  0.7433  -0.2640  0.3699
precision.2   -0.0668  0.0567  -1.1794  0.2382  -0.1779  0.0442
---
Signif. codes:  0 �***� 0.001 �**� 0.01 �*� 0.05 �.� 0.1 � � 1

Thank you very much, both of you.
Best,
JU

p.s. Wolfgang, I think I figured out what went wrong with how I specified my random effects in my previous e-mail. Specifying it as random=list(~factor(x)|Study, ~factor(x)|Id) instead of random= ~factor(x)|Study/Id generates results that makes sense to me now. Please let me know if this is correct way I should be coding.

________________________________
From: James Pustejovsky <jepusto using gmail.com>
Sent: Thursday, September 26, 2019 2:39 PM
To: Ju Lee <juhyung2 using stanford.edu>
Cc: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>; r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
Subject: Re: Covariance-variance matrix when studies share multiple treatment x control comparison

JU,

To your question about how to calculated the measure of precision: no, there's no need to create a matrix. Just a vector with the measure of precision, because it's the vector that will be used as a predictor in the meta-regression model.

James

On Thu, Sep 26, 2019 at 11:05 AM Ju Lee <juhyung2 using stanford.edu<mailto:juhyung2 using stanford.edu>> wrote:
Dear Wolfgang, James

Thank you both for your considerate suggestions.

First of all, I would like to clarify that I will be sending out another thread related to Wolfgang's comment about adding study ID to random factors as it has caused some major issues with my current analysis and I would really like second feedbacks on this matter (on my very next e-mail).

Related to James's suggestion, I will follow up on your newly published paper and apply this to my code. Since I am using variance-covariance matrix instead of normal variance (to account for shared control/treatment groups) and trying to incorporate this to modified egger's test, I am wondering if means I should be creating a diagonal matrix constituted of sqrt(1 / n1 + 1 / n2) for all inter-dependent effect sizes?

Best regards,
JU
________________________________
From: James Pustejovsky <jepusto using gmail.com<mailto:jepusto using gmail.com>>
Sent: Thursday, September 26, 2019 8:26 AM
To: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>>
Cc: Ju Lee <juhyung2 using stanford.edu<mailto:juhyung2 using stanford.edu>>; r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org> <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>>
Subject: Re: Covariance-variance matrix when studies share multiple treatment x control comparison

Ju,

Following up on Wolfgang's comment: yes, adding a measure of precision as a predictor in the multi-level/multi-variate meta-regression model should work. Dr. Belen Fernandez-Castilla has a recent paper that reports a simulation study evaluating this approach. See

Fern�ndez-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2019). Detecting selection bias in meta-analyses with multiple outcomes: A simulation study. The Journal of Experimental Education, 1�20.

However, for standardized mean differences based on simple between-group comparisons, it is better to use sqrt(1 / n1 + 1 / n2) as the measure of precision, rather than using the usual SE of d. The reason is that the SE of d is naturally correlated with d even in the absence of selective reporting, and so the type I error rate of Egger's regression test is artificially inflated if the SE is used as the predictor. Using the modified predictor as given above fixes this issue and yields a correctly calibrated test. For all the gory details, see Pustejovsky & Rodgers (2019; https://doi.org/10.1002/jrsm.1332).

It's also possible to combine all of the above with robust variance estimation, or to use a simplified model plus robust variance estimation to account for dependency between effect sizes from the same study. Melissa Rodgers and I have a working paper showing that this approach works well for meta-analyses that include studies with multiple correlated outcomes. We will be posting a pre-print of the paper soon, and I can share it on the listserv when it's available.

James

On Thu, Sep 26, 2019 at 3:12 AM Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>> wrote:
Hi Ju,

Glad to hear that you are making progress. Construction of the V matrix can be a rather tedious process and often requires quite a bit of manual work.

I have little interested in generalizing fsn() for cases where V is not diagonal, because fsn() is more of interest for historical reasons, not something I would generally use in applied work.

However, the 'Egger regression' test can be easily generalized to rma.mv<http://rma.mv>() models. Simply include a measure of the precision (e.g., the standard error) of the estimates in your model as a predictor/moderator and then you have essentially a multilevel/multivariate version thereof (you would then look at the test of the coefficient for the measure of precision, not the intercept).

I also recently heard a talk by Melissa Rodgers and James Pustejovsky (who is a frequent contributor to this mailing list) on some work in this area. Maybe he can chime in here.

Best,
Wolfgang

-----Original Message-----
From: Ju Lee [mailto:juhyung2 using stanford.edu<mailto:juhyung2 using stanford.edu>]
Sent: Thursday, 26 September, 2019 8:13
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>
Subject: Re: Covariance-variance matrix when studies share multiple treatment x control comparison

Dear Wolfgang,

I deeply appreciate your time looking into this issue, and this has been immensely helpful.
I was able to incorporate all possible inter-dependence among effect sizes by adding different layers of non-independence to our dataframe.

I manually calculated hedges'd based on based on Hedges and Olkin (1985), and it generates exactly same value as hedges' g in escalc() "SMD" function. So hopefully I am doing everything right using the equation we've discussed earlier.

I have been also wondering if it is possible to account of this variance-covariance structure that I've constructed when running publication bias analysis, for example, when using fsn() function or modified egger's regression test (looking at intercept term of residual ~ precision meta-regression using rma.mv<http://rma.mv>). I had no luck so far finding information on this, and I would appreciate if you have any suggestions related to this

Thank you for all of your valuable helps!
Best regards,
JU

[[alternative HTML version deleted]]

```