[R-meta] calculate effect size and variance for prepost proportion data

[Viechtbauer, Wolfgang (NP)]

Just a late follow-up to this:

This is actually implemented in escalc() in the metafor package. See:

https://wviechtb.github.io/metafor/reference/escalc.html#-b-measures-for-dichotomous-variables-2

References are also given there.

To illustrate:

library(metafor)

escalc(measure="MPOR", ai=30, bi=15,
                       ci= 5, di=20, digits=8)

# James' formulas give the same results
N <- 30 + 15 + 5 + 20
P7 <- (30 + 15) / N
P6 <- (30 +  5) / N
B <- 30 / N
log(P7 / (1 - P7)) - log(P6 / (1 - P6))
(P6 * (1 - P6) + P7 * (1 - P7) - 2 * (B - P6 * P7)) / (N * P6 * (1 - P6) * P7 * (1 - P7))

# alternatively, one can specify the pre-post correlation (phi coefficient)
ri <- (30*20 - 15*5) / sqrt((30+15) * (5+20) * (30+5) * (15+20))
escalc(measure="MPORM", ai=30+15, bi=5+20, ci=30+5, di=15+20, ri=ri, digits=8)

# this is useful if one just has the 'marginal' counts and one needs to
# guestimate the correlation

# show that the variance of the regular OR is the same as assuming ri=0
escalc(measure="OR", ai=30+15, bi=5+20, ci=30+5, di=15+20, digits=8)
escalc(measure="MPORM", ai=30+15, bi=5+20, ci=30+5, di=15+20, ri=0, digits=8)

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of James Pustejovsky
>Sent: Monday, 30 January, 2023 16:50
>To: Liu Sicong
>Cc: r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] calculate effect size and variance for prepost proportion
>data
>
>Hi Sicong,
>
>Responses below.
>
>James
>
>On Sun, Jan 29, 2023 at 6:30 AM Liu Sicong <64zone using gmail.com> wrote:
>
>> Hi James,
>>
>> I would like to ask two follow-up questions regarding the V(LOR) formula
>> you kindly suggested previously (the relevant part has been attached below
>> inside ###--- for your convenience)
>>
>>    - Q1: what would you suggest to do when N is different between 6th
>>    (pre )and 7th (post) grade? Perhaps use the average of Npre and Npost?
>>
>This is tricky. The statistically correct answer depends on the number of
>events among the participants who have data at both time-points, which is
>probably not reported very often in practice. As an ad hoc approach, my
>first thought would be to use a) the minimum of N-pre and N-post or b) the
>harmonic mean of N-pre and N-post, N-harmonic = 2 / (1 / N-pre + 1 /
>N-post).
>
>>    - Q2: if we assume outcomes are independent (i.e., B = P6*P7), would
>>    the situation becomes similar to computing V(LOR) for between-condition
>>    effect sizes of proportion outcomes? In such a case, would the V(LOR)
>>    formula you suggested be mathematically related to the one for
>>    between-condition effect sizes (i.e., V(LOR) = 1/A + 1/B + 1/C + 1/D, where
>>    ABCD represents the number participants in the 2*2 table)?
>
>Yes, if you assume independence, then the covariance term will drop out and
>you'll be left with
>
>V(LOR) = [P6 (1 - P6) + P7 (1 - P7)] / [N P6 (1 - P6) P7 (1 - P7)]
>= 1 / [N P6 (1 - P6)] + 1 / [N P7 (1 - P7)]
>= 1 / (N P6) + 1 / (N (1 - P6)) + 1 / (N P7) + 1 / (N (1 - P7))
>
>> Say that that the outcome is school suspension (at any time) during 6th
>> grade (pre) and 7th grade (post). Let P6 be the overall proportion of
>> students suspended during 6th grade, P7 be the overall proportion of
>> students suspended during 7th grade, and B be the proportion of students
>> suspended during both 6th and 7th grades. Let N be the total sample size
>> (which I'm assuming to be the same at both time points). The pre-post LOR is
>>
>> LOR = log[P7 / (1 - P7)] - log[P6 / (1 - P6)]
>>
>> And an estimate of its sampling variance is
>> V(LOR) = [P6 (1 - P6) + P7 (1 - P7) - 2 (B - P6 * P7)] / [N P6 (1 - P6) P7
>> (1 - P7)]
>>
>> As you can see, you'll need to know B to compute this. If this is not
>> reported, you could use a conservative estimate (i.e., a probable
>> over-estimate) of the sampling variance based on the assumption that the
>> outcomes are independent (in which case B = P6 * P7 and the last term in
>> the numerator drops out) but I'm not sure how useful that would be in your
>> application.
>>
>> ### ---
>>
>> ------------------------------------------
>> Sicong (Zone) Liu, Ph.D.
>> Research Associate
>> University of Pennsylvania
>>
>> 3620 Walnut Street,
>> Philadelphia, PA 19104-6220
>> ------------------------------------------
>>
>> *From: *James Pustejovsky <jepusto using gmail.com>
>> *Date: *Wednesday, January 4, 2023 at 11:28 PM
>> *To: *Sicong Liu <64zone using gmail.com>
>> *Cc: *"r-sig-meta-analysis using r-project.org" <
>> r-sig-meta-analysis using r-project.org>
>> *Subject: *Re: [R-meta] calculate effect size and variance for prepost
>> proportion data
>>
>> Yes, if you transform from LOR to d by taking sqrt(3 / pi) * LOR, then you
>> would multiply V(LOR) by 3 / pi.
>>
>> On Wed, Jan 4, 2023 at 7:34 PM Liu Sicong <64zone using gmail.com> wrote:
>>
>> Thank you for clarifying James!
>>
>> Just one follow-up question:
>>
>>    - If I would like to transform the V(LOR) to Cohen’s d metric, does
>>    “V(LOR) * 3/Pi” still work? Thank you!
>>
>> Cheers,
>>
>> Zone
>>
>> -------------
>> *From: *James Pustejovsky <jepusto using gmail.com>
>> *Date: *Tuesday, January 3, 2023 at 3:57 PM
>> *To: *Sicong Liu <64zone using gmail.com>
>> *Cc: *"r-sig-meta-analysis using r-project.org" <
>> r-sig-meta-analysis using r-project.org>
>> *Subject: *Re: [R-meta] calculate effect size and variance for prepost
>> proportion data
>>
>> Hi Zone,
>>
>> I have not been able to find a reference for the pre-post log odds ratio
>> in particular. I derived the formula using the delta method (same as Wei
>> and Higgins) and the properties of the multinomial distribution.
>>
>> Perhaps others on the list know of a reference?
>>
>> James
>>
>> On Tue, Jan 3, 2023 at 2:13 PM Liu Sicong <64zone using gmail.com> wrote:
>>
>> Happy 2023 and thank you for your response, James!
>>
>> I wonder if you could point me to the reference of the formulas raised,
>> especially the V(LOR) one? I checked Wei and Higgins (2013) but did not
>> find such a formula explicitly expressed in the paper. Perhaps the V(LOR)
>> is derived from their general method? Please let me know.
>>
>> Cheers,
>> Zone
>>
>> -------------
>>
>> *From: *James Pustejovsky <jepusto using gmail.com>
>> *Date: *Tuesday, January 3, 2023 at 10:43 AM
>> *To: *Sicong Liu <64zone using gmail.com>
>> *Cc: *"r-sig-meta-analysis using r-project.org" <
>> r-sig-meta-analysis using r-project.org>
>> *Subject: *Re: [R-meta] calculate effect size and variance for prepost
>> proportion data
>>
>> Hi Zone,
>>
>> I think it is less common to use pre-post effect size measures with binary
>> outcomes. In principle, it can be done, but my sense is that there is less
>> benefit (in terms of precision improvement) from using a binary pre-test
>> than there is from accounting for pre-tests with continuous outcomes.
>>
>> Wei and Higgins (2013; https://doi.org/10.1002/sim.5679) discuss the
>> covariance between log odds ratios computed for different binary outcomes,
>> which is closely related to the case you're looking at. In order to get an
>> accurate estimate of the sampling variance of the pre-post log odds ratio,
>> you will need to know the correlation between the pre-test outcome and the
>> post-test outcome or, equivalently, the number of participants with the
>> positive outcome at both pre-test and post-test.
>>
>> Say that that the outcome is school suspension (at any time) during 6th
>> grade (pre) and 7th grade (post). Let P6 be the overall proportion of
>> students suspended during 6th grade, P7 be the overall proportion of
>> students suspended during 7th grade, and B be the proportion of students
>> suspended during both 6th and 7th grades. Let N be the total sample size
>> (which I'm assuming to be the same at both time points). The pre-post LOR is
>>
>> LOR = log[P7 / (1 - P7)] - log[P6 / (1 - P6)]
>>
>> And an estimate of its sampling variance is
>>
>> V(LOR) = [P6 (1 - P6) + P7 (1 - P7) - 2 (B - P6 * P7)] / [N P6 (1 - P6) P7
>> (1 - P7)]
>>
>> As you can see, you'll need to know B to compute this. If this is not
>> reported, you could use a conservative estimate (i.e., a probable
>> over-estimate) of the sampling variance based on the assumption that the
>> outcomes are independent (in which case B = P6 * P7 and the last term in
>> the numerator drops out) but I'm not sure how useful that would be in your
>> application.
>>
>> James
>>
>> On Mon, Jan 2, 2023 at 7:40 AM Liu Sicong <64zone using gmail.com> wrote:
>>
>> Happy 2023 All!
>>
>> I have some prepost proportion data. For instance, some clinical trials
>> may intervene on patients’ vaccine uptake and report the proportion of
>> patients who received the vaccine both prior to and after interventions. So
>> I may have the following data
>>
>>   *   Outcomes in proportion: p_control_pre, p_control_post,
>> p_experiment_pre, p_experiment_post
>>   *   Sample sizes: n_control_pre, n_control_post, n_experiment_pre,
>> n_experiment_post
>>
>> I am clear about how to calculate between-condition effect sizes and
>> variances in the following manner. For instance, those for comparing the
>> conditions at posttest would be:
>>
>>   *   Effect size: ln((p_experiment_post/(1 -
>> p_experiment_post))/(p_control_post/(1 - p_control_post)))
>>   *   Variance of effect size: 1/(n_experiement_post*p_experiment_post) +
>> 1/(n_experiement_post*(1-p_experiment_post)) +
>> 1/(n_control_post*p_control_post) + 1/(n_control_post*(1-p_control_post))
>>
>> My question is about how to calculate the effect size and its variance
>> when I am also interested in within-condition growth. For instance, how to
>> represent the prepost growth due to vaccination intervention for the
>> experimental group? Perhaps even before asking this question, would it be
>> reasonable to attempt the computation of such effect sizes and variances?
>> Thank you very much!
>>
>> Best regards,
>> Sicong (Zone)
>>
>> ------------------------------------------
>> Sicong (Zone) Liu, Ph.D.
>> Research Associate
>> University of Pennsylvania
>>
>> 3620 Walnut Street,
>> Philadelphia, PA 19104-6220
>> ------------------------------------------