[R-meta] Question regarding three-level metaanalysis of proportions

[Viechtbauer, Wolfgang (SP)]

Thanks. Ok, the coding looks fine. I was worried that effect_id was coded like this:

StudyID         EffectID
DoiNR_study1    1
DoiNR_study1    2
DoiNR_study3    1
DoiNR_study4    1
DoiNR_study4    2
DoiNR_study4    3

in which case random = list(~ 1 | effect_id, ~ 1 | doi) would not be appropriate and instead random = ~ 1 | doi/effect_id would have to be used. But that's not the issue here.

Based on the results you posted, the estimated (back-transformed) proportions are:

> transf.iarcsin(.5027)
[1] 0.2321247
> transf.iarcsin(.5027-.0443)
[1] 0.1958185
> transf.iarcsin(.5027-.1458)
[1] 0.1220603

or what you should do is use:

predict(ies, newmods=c(0,0), transf=transf.iarcsin, digits=2)
predict(ies, newmods=c(1,0), transf=transf.iarcsin, digits=2)
predict(ies, newmods=c(0,1), transf=transf.iarcsin, digits=2)

to get these values plus CIs/PIs.

So the estimates for timepoints 1 and 2 are very similar, but the .12 does differ a bit more from the .19 you find when just running a random-effects model at each timepoint. There are several possible reasons for this:

1) In the multilevel model, the true outcomes (i.e., the arcsine-transformed values of the true proportions) are allowed to be correlated. As a result, results for timepoints 1 and 2 can also have an impact on results for timepoint 3.

2) When you run three separate RE models, you are automatically allowing tau^2 to differ at the three timepoints. The multilevel model you specified on the other hand assumes that the two variance components (sigma^2.1 and sigma^2.2) are the same at all three timepoints. You could also try the following. First run:

random = ~ factor(timepoint) | doi, struct="CS"

to ensure that the results will be the same. This uses a different parameterization of multilevel model, namely in terms of the total variance (sigma^2.1 + sigma^2.2) and the correlation in the true outcomes (rho). Also, instead of effect_id, you can use 'timepoint' here. This assumes that, within each study, there is only a single row per timepoint. If that is not the case, then things will be different, but let me assume this now.

Next, set struct="HCS". This then allows the total variance to differ at the three timepoints. This is now one step closer to running three separate models (still not the same due to 1) above, but more similar). With three timepoints, you could even try struct="UN", which also allows the correlation between pairs of timepoints to differ (since the correlation between 1 and 2, 1 and 3, and 2 and 3 might differ).

One last point: Do you actually have longitudinal data? That is, are the proportions at the three timepoints within a study based on the same sample? Then you cannot assume that the sampling errors of the (transformed) proportions are independent of each other. To take this properly into consideration, you should look into:

help(dat.ishak2007)

and

help(dat.fine1993)

as examples of longitudinal meta-analyses.

And if so, one would have to further distinguish between the scenario where the event of interest (being a 'postitive case' in your data) can happen multiple times / intermittently to the same individual (e.g., the same subject can be a positive at timepoints 1 and 3 but not at 2) or where a subject, once found to be a postitive case, stays so. In that latter case, one can compute the proportions in different ways: 1) if positive at timepoint 1, then the subject is automatically counted as postitive for timepoints 2 and 3 and still counted for the denominator or 2) if positive at timepoint 1, then the subject is removed from further calculations of the proportions at timepoints 2 and 3.

Best,
Wolfgang

>-----Original Message-----
>From: simeon.zuercher using upd.unibe.ch [mailto:simeon.zuercher using upd.unibe.ch]
>Sent: Wednesday, 23 June, 2021 12:52
>To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
>Subject: AW: [R-meta] Question regarding three-level metaanalysis of proportions
>
>Dear Wolfgang,
>Tank you very much for the fast answer.
>
>I have a studyID which is the same Doi for a single study and unique numbers for
>effectIDs. The data is structured as follows (example data):
>
>StudyID         EffectID
>DoiNR_study1    1
>DoiNR_study1    2
>DoiNR_study3    3
>DoiNR_study4    4
>DoiNR_study4    5
>DoiNR_study4    6
>
>Estimating the proportions for each timepoint using the arcsine transformation
>(instead double arcsine) yields the same results. Time 1 = 0.22, Time 2 =  0.17,
>Time 3 =0.19
>
>I used now used the arcsine transformation to calculate the model
>
>ies <- escalc(xi=postitive_cases, ni= sample_size, data = ies, measure = "PAS")
>estimates <- rma.mv(yi, vi, mods = ~ timepoint, random = list(~ 1 | effect_id,
>~ 1 | doi), tdist = TRUE, data = ies, method = "REML")
>
>Output:
>
>Multivariate Meta-Analysis Model (k = 99; method: REML)
>Variance Components:
>            estim    sqrt  nlvls  fixed     factor
>sigma^2.1  0.0069  0.0833     99     no  effect_id
>sigma^2.2  0.0216  0.1469     47     no        doi
>
>Test for Residual Heterogeneity:
>QE(df = 96) = 1310.2448, p-val < .0001
>Test of Moderators (coefficients 2:3):
>F(df1 = 2, df2 = 96) = 3.8752, p-val = 0.0241
>
>Model Results:
>
>                           estimate       se     tval    pval    ci.lb    ci.ub
>intrcpt(timepoint 1)         0.5027   0.0272  18.4506  <.0001   0.4486   0.5568
>timepoint 2                 -0.0443   0.0459  -0.9632  0.3379  -0.1355   0.0469
>timepoint 3)                -0.1458   0.0538  -2.7092  0.0080  -0.2526  -0.0390
>
>backtransformation of estimates yield
>sin(0.5027)^2 = 0.23,
>sin(-0.0443)^2 = 0.0020,
>sin(-0.1458)^2 = 0.021
>
>I think this is much more plausible. F.i. the difference between timepoint 1 and 3
>is now  -0.02 (or two percentage points).
>
>What do you think. Does it look plausible to you?
>
>again, many thanks,
>Simeon
>
>________________________________________
>Von: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Gesendet: Mittwoch, 23. Juni 2021 10:57:48
>An: Zürcher, Simeon (UPD); r-sig-meta-analysis using r-project.org
>Betreff: RE: [R-meta] Question regarding three-level metaanalysis of proportions
>
>Dear Simeon,
>
>I generally would avoid meta-analyzing proportions directly. Instead of the double
>arcsine transformation (which indeed leads to issues with the back-
>transformation), you could use the logit transformation or the arcsine square root
>transformation, both of which are easy to back-transform.
>
>This aside, for a more direct comparison of the two modeling approaches, I would
>suggest to use the same transformation. Also, can you show how 'effect_id' is
>coded? In other words, is it a unique value for every row of the dataset? It might
>also help if you would post the actual output from the model.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of simeon.zuercher using upd.unibe.ch
>>Sent: Wednesday, 23 June, 2021 10:32
>>To: lists using dewey.myzen.co.uk; r-sig-meta-analysis using r-project.org
>>Subject: Re: [R-meta] Question regarding three-level metaanalysis of proportions
>>
>>Dear all,
>>
>>Only recently, I had a question regarding three-level meta-analysis of
>proportions
>>where we look at neurological complications after some infectious diseases which
>>was kindly answered (Thank you again). We effect sizes are dependent since some
>>studies report several effect sizes. It's a large dataset with over 150 effect
>>sizes (effect_id) that are nested within 60 studies (doi).
>>
>>Based on the advice we did not transform proportions with double arcsine for the
>>meta-regression since a back-transformation after model estimation is not
>straight
>>forward. However, we have now the following issue: I have performed a meta-
>>regression on timepoint as moderator (timepoint = factor variable including three
>>timepoints 1,2 and 3).
>>
>>The meta-analysis (with double arcsine transformation) gave the following
>>proportion for each timepoint separately: Time 1 = 0.22, Time 2 =  0.17, Time 3 =
>>0.19
>>
>>However, if I run a meta-regression on proportions to see whether the timepoints
>>differ I get very strange results.
>>
>>result_0 <- rma.mv(yi, vi, random = list(~ 1 | effect_id, ~ 1 | doi),
>>
>>                  mods = ~ timepoint, tdist = TRUE, data = data, method = "REML")
>>
>>Intercept (timepoint 1): 0.24 (seems plausible)
>>
>>Time 2: -0.032 (seems also plausible),
>>
>>Time 3: -0.15 (is not plausible)
>>
>>The estimate for Time 3 vs. Time 1 is very strange. Even if I remove the
>outliers,
>>I got such an extreme result. Is there something wrong in this code? Maybe the
>>definition of the random effects?
>>
>>Many thanks for your further help!
>>
>>Kind regars,
>>
>>Simeon