[R-meta] I have any problems with meta-analysis of proportions
Dr. Gerta Rücker
ruecker @end|ng |rom |mb|@un|-|re|burg@de
Mon May 24 20:23:45 CEST 2021
Dear Martin,
Assume on the logit scale we have some estimated average logit
proportion, say logit \hat p. Based on the random effects model, the
distribution of true proportions in the studies is assumed to be normal
with mean \hat p and standard deviation tau. You may look at the interval
[logit \hat p - tau, logit \hat p + tau]
(still on the logit scale) and transform this to the natural scale by
using the inverse of the logit, which is the expit function:
expit(x) = exp(x)/(1 + exp(x))
You obtain the transformed interval
[\hat p - expit(tau), \hat p + expit(tau)] = [\hat p - exp(tau)/(1 +
exp(tau)), \hat p + exp(tau)/(1 + exp(tau))]
This is an interval in which you would expect about 68% of estimates to lie.
The prediction interval is defined differently, it uses a 95% quantile
of a t- distribution, see help(metaprop) and Higgins JPT, Thompson SG,
Spiegelhalter DJ (2009): A re-evaluation of random-effects
meta-analysis. Journal of the Royal Statistical Society: Series A, 172,
137–59.
How to interpret your prediction interval? Difficult to say without
knowing your research question and your prediction interval ...
Generally, the prediction interval describes a region in which 95% of
estimates are expected. For proportions, heterogeneity is usually large,
it may well be that the prediction interval goes from 1% to 70% or so. I
don't know your numbers, but probably everybody would agree that this
would be large.
See also
https://bmcmedresmethodol.biomedcentral.com/track/pdf/10.1186/1471-2288-8-79.pdf
, page 7 bottom left there.
Best,
Gerta
Am 24.05.2021 um 19:56 schrieb Martin Lobo:
> Thanks Gerta,
> and how do I convert the tau2 to a scale of proportions to evaluate
> heterogeneity? I am having a hard time assessing / reporting
> heterogeneity in this meta-analysis of proportions.
> how should you interpret the prediction interval to assess herogenicity.
>
> thank you
>
>
>
> */
> /*
> */Lorenzo Martín Lobo /**/^MTSAC, FACC, FESC /*
> /*Especialista Jerarquizado en Cardiología*/
> /*/*Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital
> Militar Campo de Mayo.*/
> */
> /*/*Jefe de Cardiología *//*Hospital Militar Campo de Mayo*/
> */
> /*Ex Jefe de Unidad Coronaria *//*Hospital Militar Campo de Mayo*/
> /
> /*Miembro Titular de la Sociedad Argentina de Cardiología*/
> /
> /
> /*Fellow American College of Cardiology*/
> /
> /
> /*Fellow European Society of Cardiology*/
> /
> /
> /*/*Ex Miembro del Area de Investigación de la SAC*/*/
> /
> /*Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC*/
> /*Miembro Asesor /*del Consejo de Aterosclerosis y Trombosis de la SAC*/*/
> /*/
> /*/
> /*Ex Director del Consejo de Epidemiología y Prevención Cardiovascular
> de la SAC*/
> /*/
> /*/
> /*/
> /*//*/
> /*/
> /*Miembro Asesor del Consejo de Epidemiología y Prevención
> Cardiovascular de la SAC*/
> /*/
> /*/
> /*/
> /*/
> /*Experto en Lipidos de la Sociedad Argentina de Lipidos.*/
> /*Miembro de la Sociedad Argentina de Lipidos.*/
> /
> /*Instructor de ACLS de la American Heart Association*/
> /
>
>
> ------------------------------------------------------------------------
> *De:* Dr. Gerta Rücker <ruecker using imbi.uni-freiburg.de>
> *Enviado:* lunes, 24 de mayo de 2021 14:03
> *Para:* Martin Lobo <mlobo4370 using hotmail.com>; Lukasz Stasielowicz
> <lukasz.stasielowicz using uni-osnabrueck.de>;
> r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
> *Asunto:* Re: [R-meta] I have any problems with meta-analysis of
> proportions
> Dear Martin,
>
> The print output of metaprop() gives you the information which
> transformation was used. The default method is the logit transformation
> (internally, the rma.glmm function from R package metafor is called to
> fit a generalized mixed logistic regression model). As you did not
> change the sm argument, this was indeed used, baset on the logit
> transformation. This means that your value of tau is on the logit scale.
> If you want to see how large heterogeneity is on the natural scale, you
> may plot a forest plot with a prediction interval which can be done using
>
> forest(ABA3L, prediction = TRUE)
>
> Best,
>
> Gerta
>
> Am 24.05.2021 um 18:19 schrieb Martin Lobo:
> > Hello, I come back.
> > Lukasz,
> > How can I know if the tau is transformed to be able to make the
> interpretation as you suggested to me. I am using the metaprop command
> from the meta package with its default configuration. Thanks.
> >
> > Ex:
> > ABA3L<-metaprop(EventNumber,TotalNumber, data = data)
> > regards
> >
> >
> >
> >
> > Lorenzo Martín Lobo MTSAC, FACC, FESC
> > Especialista Jerarquizado en Cardiología
> > Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital
> Militar Campo de Mayo.
> > Jefe de Cardiología Hospital Militar Campo de Mayo
> > Ex Jefe de Unidad Coronaria Hospital Militar Campo de Mayo
> > Miembro Titular de la Sociedad Argentina de Cardiología
> > Fellow American College of Cardiology
> > Fellow European Society of Cardiology
> > Ex Miembro del Area de Investigación de la SAC
> > Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC
> > Miembro Asesor del Consejo de Aterosclerosis y Trombosis de la SAC
> > Ex Director del Consejo de Epidemiología y Prevención Cardiovascular
> de la SAC
> >
> > Miembro Asesor del Consejo de Epidemiología y Prevención
> Cardiovascular de la SAC
> >
> >
> > Experto en Lipidos de la Sociedad Argentina de Lipidos.
> > Miembro de la Sociedad Argentina de Lipidos.
> > Instructor de ACLS de la American Heart Association
> >
> >
> > ________________________________
> > De: Lukasz Stasielowicz <lukasz.stasielowicz using uni-osnabrueck.de>
> > Enviado: viernes, 9 de abril de 2021 17:07
> > Para: r-sig-meta-analysis using r-project.org
> <r-sig-meta-analysis using r-project.org>
> > Cc: mlobo4370 using hotmail.com <mlobo4370 using hotmail.com>
> > Asunto: Re: I have any problems with meta-analysis of proportions
> >
> > Dear Martin,
> >
> > if tau-squared equals .06 then tau equals sqrt(.06) = .24. It would be
> > huge in the context of proportions. To illustrate, if the mean
> > proportion is .34 then it would mean that proportions varying between
> > .10 and .58 could be considered normal (tau can be regarded as the true
> > standard deviation of effect sizes).
> >
> > I wonder, whether the reported tau-squared value is really on the
> > proportion scale. You are probably using some kind of transformation in
> > your meta-analysis rather than using raw proportions. In many
> > statistical packages the estimates are not automatically
> > back-transformed in the output. In this case one would need to
> > back-transform the tau-squared estimate manually. Only then we can
> > interpret the tau value on the proportion scale.
> >
> > The reported range of proportions (.28 to .59) seems to indicate that
> > there is some heterogeneity but it could be a premature conclusion.
> > Consider a scenario where one proportion equals .28 and all other
> > proportions are close to .59. The heterogeneity could be attributed to
> > one outlier (.28). In order to rule out such scenarios you could plot
> > the distribution of the observed proportions (e.g., histogram, boxplot
> > or violin plot).
> >
> >
> > Best wishes,
> > Lukasz
> > --
> > Lukasz Stasielowicz
> > Osnabrück University
> > Institute for Psychology
> > Research methods, psychological assessment, and evaluation
> > Seminarstraße 20
> > 49074 Osnabrück (Germany)
> >
> > Am 09.04.2021 um 11:35 schrieb
> r-sig-meta-analysis-request using r-project.org:
> >> Send R-sig-meta-analysis mailing list submissions to
> >> r-sig-meta-analysis using r-project.org
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161147968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dSVMVxbhnc4iQgQSh2Al3%2FxOrZ5n5vXPF4JJiNZD6DI%3D&reserved=0
> >> or, via email, send a message with subject or body 'help' to
> >> r-sig-meta-analysis-request using r-project.org
> >>
> >> You can reach the person managing the list at
> >> r-sig-meta-analysis-owner using r-project.org
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of R-sig-meta-analysis digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >> 1. Re: I have any problems with meta-analysis of proportions
> >> (Martin Lobo)
> >> 2. Multivariate data: RVE imputing covariance matrices
> >> (Bernard Fernou)
> >>
> >> ----------------------------------------------------------------------
> >>
> >> Message: 1
> >> Date: Thu, 8 Apr 2021 16:15:12 +0000
> >> From: Martin Lobo <mlobo4370 using hotmail.com>
> >> To: Michael Dewey <lists using dewey.myzen.co.uk>,
> >> "r-sig-meta-analysis using r-project.org"
> >> <r-sig-meta-analysis using r-project.org>, Nicky Welton
> >> <Nicky.Welton using bristol.ac.uk>,
> "lukasz.stasielowicz using uni-osnabrueck.de"
> >> <lukasz.stasielowicz using uni-osnabrueck.de>
> >> Subject: Re: [R-meta] I have any problems with meta-analysis of
> >> proportions
> >> Message-ID:
> >>
> <MN2PR20MB28145EC2D2B12A6BA4AC7957AC749 using MN2PR20MB2814.namprd20.prod.outlook.com>
> >>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> thank you all, you have clarified a lot for me.
> >> some clarifications for you
> >>
> >> Michael,
> >> In this quiestion:
> >> If so, how do I describe the methodological part, what guidelines
> and quality scales should I use (PRISMA, STROBE, COCHRANE, NOS, JADAD?).
> >>
> >> I have read that for meta analysis of observational studies the
> STROBE guide should be used. Is it the same to use the PRISMA gui as
> the STROBE in this type of meta analysis?
> >>
> >>
> >> Lukasz,
> >>
> >> In some of my examples the I2 is 97%, T2 (square tau)0.06 (0
> <0.01), in this case how would you consider heterogenicity?
> >> the proportion estimates vary from 0.28 to 0.59.
> >>
> >> Thank's for your help
> >>
> >> Regards
> >>
> >>
> >>
> >>
> >> Lorenzo Martín Lobo MTSAC, FACC, FESC
> >> Especialista Jerarquizado en Cardiología
> >> Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital
> Militar Campo de Mayo.
> >> Jefe de Cardiología Hospital Militar Campo de Mayo
> >> Ex Jefe de Unidad Coronaria Hospital Militar Campo de Mayo
> >> Miembro Titular de la Sociedad Argentina de Cardiología
> >> Fellow American College of Cardiology
> >> Fellow European Society of Cardiology
> >> Ex Miembro del Area de Investigación de la SAC
> >> Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC
> >> Miembro Asesor del Consejo de Aterosclerosis y Trombosis de la SAC
> >> Ex Director del Consejo de Epidemiología y Prevención
> Cardiovascular de la SAC
> >>
> >> Miembro Asesor del Consejo de Epidemiología y Prevención
> Cardiovascular de la SAC
> >>
> >>
> >> Experto en Lipidos de la Sociedad Argentina de Lipidos.
> >> Miembro de la Sociedad Argentina de Lipidos.
> >> Instructor de ACLS de la American Heart Association
> >>
> >>
> >> ________________________________
> >> De: Michael Dewey <lists using dewey.myzen.co.uk>
> >> Enviado: martes, 6 de abril de 2021 05:48
> >> Para: Martin Lobo <mlobo4370 using hotmail.com>;
> r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
> >> Asunto: Re: [R-meta] I have any problems with meta-analysis of
> proportions
> >>
> >> Comments in-line
> >>
> >> On 05/04/2021 17:32, Martin Lobo wrote:
> >>> Hi everyone,
> >>>
> >>> I performed a systematic review on the persistence of some drugs.
> >>> I found 30 randomized clinical trials and 10 observational studies.
> >>> Although I understand that they should not be meta analyzed
> together (I should stratify them or analyze them separately),
> Actually, of the RCTs, I only use the active drug arm, so I think that
> by breaking the branching, maybe I could take all the data as
> observational.
> >> Yes, you now have a set of observational studies if you only take one
> >> arm from the trials.
> >>
> >>> Is this correct ?
> >>> If so, how do I describe the methodological part, what guidelines
> and quality scales should I use (PRISMA, STROBE, COCHRANE, NOS, JADAD?).
> >>>
> >> It is still a meta-analysis so use PRISMA
> >>
> >>> On the other hand, I have never performed meta-analysis of
> proportions, and I am having too much heterogeneity I2 97%. How could
> I control this? The studies are of good quality. I use the metaprop
> function.
> >>>
> >> In a meta-analysis of observational studies high heterogeneity is
> almost
> >> ineitable.
> >>
> >>> Than's
> >>>
> >>>
> >>>
> >>>
> >>> Lorenzo Mart�n Lobo MTSAC, FACC, FESC
> >>> Especialista Jerarquizado en Cardiolog�a
> >>> Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital
> Militar Campo de Mayo.
> >>> Jefe de Cardiolog�a Hospital Militar Campo de Mayo
> >>> Ex Jefe de Unidad Coronaria Hospital Militar Campo de Mayo
> >>> Miembro Titular de la Sociedad Argentina de Cardiolog�a
> >>> Fellow American College of Cardiology
> >>> Fellow European Society of Cardiology
> >>> Ex Miembro del Area de Investigaci�n de la SAC
> >>> Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC
> >>> Miembro Asesor del Consejo de Aterosclerosis y Trombosis de la SAC
> >>> Ex Director del Consejo de Epidemiolog�a y Prevenci�n
> Cardiovascular de la SAC
> >>>
> >>> Miembro Asesor del Consejo de Epidemiolog�a y Prevenci�n
> Cardiovascular de la SAC
> >>>
> >>>
> >>> Experto en Lipidos de la Sociedad Argentina de Lipidos.
> >>> Miembro de la Sociedad Argentina de Lipidos.
> >>> Instructor de ACLS de la American Heart Association
> >>>
> >>>
> >>> ________________________________
> >>> De: R-sig-meta-analysis
> <r-sig-meta-analysis-bounces using r-project.org> en nombre de
> r-sig-meta-analysis-request using r-project.org
> <r-sig-meta-analysis-request using r-project.org>
> >>> Enviado: s�bado, 31 de octubre de 2020 08:05
> >>> Para: r-sig-meta-analysis using r-project.org
> <r-sig-meta-analysis using r-project.org>
> >>> Asunto: R-sig-meta-analysis Digest, Vol 41, Issue 19
> >>>
> >>> Send R-sig-meta-analysis mailing list submissions to
> >>> r-sig-meta-analysis using r-project.org
> >>>
> >>> To subscribe or unsubscribe via the World Wide Web, visit
> >>>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161147968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dSVMVxbhnc4iQgQSh2Al3%2FxOrZ5n5vXPF4JJiNZD6DI%3D&reserved=0
> >>> or, via email, send a message with subject or body 'help' to
> >>> r-sig-meta-analysis-request using r-project.org
> >>>
> >>> You can reach the person managing the list at
> >>> r-sig-meta-analysis-owner using r-project.org
> >>>
> >>> When replying, please edit your Subject line so it is more specific
> >>> than "Re: Contents of R-sig-meta-analysis digest..."
> >>>
> >>>
> >>> Today's Topics:
> >>>
> >>> 1. Re: GOSH plots for multi-level meta (rma.mv) (Hellen Mirr)
> >>>
> >>> ----------------------------------------------------------------------
> >>>
> >>> Message: 1
> >>> Date: Fri, 30 Oct 2020 13:03:34 +0000
> >>> From: Hellen Mirr <hellenmir554 using gmail.com>
> >>> To: "Viechtbauer, Wolfgang (SP)"
> >>> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> >>> Cc: "r-sig-meta-analysis using r-project.org"
> >>> <r-sig-meta-analysis using r-project.org>
> >>> Subject: Re: [R-meta] GOSH plots for multi-level meta (rma.mv)
> >>> Message-ID:
> >>> <CAF6nRJqkgQ4_vkF0sdf=_anW2Etp2snB14F-v0ot8rZ9JFaXGQ using mail.gmail.com>
> >>> Content-Type: text/plain; charset="utf-8"
> >>>
> >>> Dear Wolfgang,
> >>>
> >>> Thank you very much for your clear explanation.
> >>>
> >>> Best,
> >>> Hellen
> >>>
> >>> On Fri, Oct 30, 2020 at 11:48 AM Viechtbauer, Wolfgang (SP) <
> >>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >>>
> >>>> Dear Hellen,
> >>>>
> >>>> This is currently not implemented in metafor. In principle, the
> idea of a
> >>>> GOSH plot does generalize to more complex models although one
> needs to
> >>>> think about whether one would want to create subsets based on the
> indiviual
> >>>> estimates or based on some higher-level grouping variable. For
> example,
> >>>> suppose we have a multilevel structure such as:
> >>>>
> >>>> study esid yi vi
> >>>> ------------------
> >>>> 1 1 . .
> >>>> 1 2 . .
> >>>> 2 1 . .
> >>>> 3 1 . .
> >>>> 3 2 . .
> >>>> 3 3 . .
> >>>> 4 1 . .
> >>>>
> >>>> So, what are, for example, then the subsets of size 2? Are they
> based just
> >>>> on the rows? Then the estimates in row 1 and 2 would be one such
> subset. Or
> >>>> does one base the subsets on the studies? Then rows 1, 2, 3
> (i.e., studies
> >>>> 1 and 2) would be such a subset.
> >>>>
> >>>> This could all be implemented (just like cooks.distance() and
> rstudent()
> >>>> allow for the optional specification of a clustering variable), but I
> >>>> haven't done this.
> >>>>
> >>>> Aside from that: Fitting rma.mv models can take a bit of time.
> Doing this
> >>>> 1000's of times (based on all possible subsets) could take a LONG
> time.
> >>>>
> >>>> Best,
> >>>> Wolfgang
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: R-sig-meta-analysis [mailto:
> >>>> r-sig-meta-analysis-bounces using r-project.org]
> >>>>> On Behalf Of Hellen Mirr
> >>>>> Sent: Friday, 30 October, 2020 12:12
> >>>>> To: r-sig-meta-analysis using r-project.org
> >>>>> Subject: [R-meta] GOSH plots for multi-level meta (rma.mv)
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> Apologies if this has already been answered, as I could not find any
> >>>>> threads on it.
> >>>>> I was wondering whether it is possible to create a GOSH plot for a
> >>>>> multi-level meta analysis that is an rma.mv object, and how I
> would go
> >>>>> about that.
> >>>>>
> >>> [[elided Hotmail spam]]
> >>>>> Best wishes
> >>>>> Hellen
> >>> [[alternative HTML version deleted]]
> >>>
> >>>
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> Subject: Digest Footer
> >>>
> >>> _______________________________________________
> >>> R-sig-meta-analysis mailing list
> >>> R-sig-meta-analysis using r-project.org
> >>>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161157958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YgI%2BT5fhl925jleGSYW%2BgqSBoSW0bNrYUbvKMVw%2BT2A%3D&reserved=0
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> End of R-sig-meta-analysis Digest, Vol 41, Issue 19
> >>> ***************************************************
> >>>
> >>> [[alternative HTML version deleted]]
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> R-sig-meta-analysis mailing list
> >>> R-sig-meta-analysis using r-project.org
> >>>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161157958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YgI%2BT5fhl925jleGSYW%2BgqSBoSW0bNrYUbvKMVw%2BT2A%3D&reserved=0
> >>>
> >> --
> >> Michael
> >>
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161157958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yTSuIQP7jtPqNyzrBu7wUZSjMBxM%2FaFfAbl%2FR0ASpb0%3D&reserved=0
> >>
> >> [[alternative HTML version deleted]]
> >>
> >>
> >>
> >> ------------------------------
> >>
> >> Message: 2
> >> Date: Fri, 9 Apr 2021 11:35:36 +0200
> >> From: Bernard Fernou <bernard.fernou using gmail.com>
> >> To: r-sig-meta-analysis using r-project.org
> >> Subject: [R-meta] Multivariate data: RVE imputing covariance matrices
> >> Message-ID:
> >> <CA+zsPoRB_J2cWuxFnJ+3u6joZmzXQfHn0VxMm3X0HmcuBpPhYw using mail.gmail.com>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Dear all
> >>
> >> We are conducting a meta-analysis of observational studies in which
> several
> >> studies have mulitple effect sizes.
> >>
> >> *Description of our data*
> >>
> >> So that you have a reproducible example, I have adapted the
> "SATcoaching"
> >> data from the "clubSandwich" to mimic our situation (the code is
> provided
> >> below).
> >>
> >> In our meta-analysis, we collected data about 60 studies (~100 effect
> >> sizes) that used Pearson's correlation to measure the effect size
> of the
> >> association between two numeric constructs. Our "independent variable"
> >> could be measured with 4 different methods and our "outcome" could be
> >> measured with 8 different methods.
> >>
> >> Some studies reported multiple effect sizes. This comes from the
> fact (i)
> >> studies included multiple measures of the outcome or (ii) studies
> included
> >> multiple measures of the independent variable (no study combined
> multiple
> >> measures of the independent variable with multiple measures of the
> >> outcome).
> >>
> >> Unfortunately, we do not have the correlation matrix between the
> >> different measures
> >> of the outcome/independent variable.
> >>
> >> *Objective of our meta-analysis*
> >>
> >> We are interested in generating:
> >>
> >> 1) a pooled effect size across measures
> >>
> >> 2) a pooled effect size for each combination of outcome and independent
> >> variable measures
> >>
> >>
> >> *Approach 1 (standard RVE)*
> >>
> >> Initially, our approach was to use a robust variance estimator
> using the
> >> robumeta package to take into account the dependency between our effect
> >> sizes.
> >>
> >> #main analysis:
> >>
> >> robumeta::robu(z ~ 1,
> >> data = SATcoaching_order ,
> >> studynum = study,
> >> var.eff.size = Vz,
> >> modelweights = "CORR",
> >> small = TRUE)
> >>
> >> #For the moderation assessing the pooled effect size for each
> combination
> >> of outcome/independent variable measure, we created a variable
> combining
> >> all observed combination of outcome measures and independent variable
> >> measures (labelled "out_var")
> >>
> >> robumeta::robu(z ~ -1 + out_var,
> >> data = SATcoaching_order,
> >> studynum = study,
> >> var.eff.size = Vz,
> >> modelweights = "CORR",
> >> small = TRUE)
> >>
> >>
> >> *Approach 2 (CSE)*
> >>
> >> I have recently read with a lot of interest the preprint of Profs
> >> Pustejovsky and Tipton (entitled "Meta-Analysis with Robust Variance
> >> Estimation: Expanding the Range of Working Models").
> >>
> >> As they described, we assume that we have a within-study
> heterogeneity in
> >> true effect size according to the outcome/independent variable measures
> >> (empirically, we found that some combination have a very low
> inconsistency
> >> ~20% while others have a very high 90%)
> >>
> >> #This is how I have implemented this approach
> >>
> >> V_list <- impute_covariance_matrix(vi = SATcoaching_order$Vz,
> >>
> >> cluster =
> SATcoaching_order$study,
> >>
> >> r = 0.7,
> >>
> >> return_list = FALSE,
> >>
> >> smooth_vi = TRUE,
> >>
> >> subgroup =
> SATcoaching_order$out_var)
> >>
> >>
> >>
> >> #primary analysis with all outcome/independent variable measures
> combined)
> >>
> >>
> >>
> >> res <- metafor::rma.mv(yi = z,
> >>
> >> V = V_list,
> >>
> >> data = SATcoaching_order,
> >>
> >> random = list(~ out_var | study),
> >>
> >> struct = "DIAG",
> >>
> >> sparse = TRUE)
> >>
> >>
> >> coef_test(res, vcov = "CR2")
> >>
> >>
> >>
> >> #moderation analysis with all combination possible
> >>
> >> resmod <- metafor::rma.mv(yi = z,
> >>
> >> V = V_list,
> >>
> >> mods = ~ out_var - 1,
> >>
> >> data = SATcoaching_order,
> >>
> >> random = list(~ out_var | study),
> >>
> >> struct = "DIAG",
> >>
> >> sparse = TRUE)
> >>
> >>
> >> coef_test(resmod, vcov = "CR2")
> >>
> >>
> >> #In moderation model, some of the moderation modalities produced
> NaN when
> >> requesting robust SE (but not model-based SE).
> >>
> >>
> >> *Approach 3 (CHE)*
> >>
> >> Because the previous model did not estimate all modalities using robust
> >> standard errors, we have tried a simpler model for a sensitivity
> analysis
> >>
> >> V_list2 <- impute_covariance_matrix(vi = SATcoaching_order$Vz,
> >>
> >> cluster =
> SATcoaching_order$study,
> >>
> >> r = 0.7,
> >>
> >> return_list = FALSE,
> >>
> >> smooth_vi = TRUE)
> >>
> >> ###subgroup = SATcoaching_order$out_var
> >> )###
> >>
> >>
> >> resmod2 <- metafor::rma.mv(yi = z,
> >>
> >> V = V_list2,
> >>
> >> mods = ~ out_var - 1,
> >>
> >> data = SATcoaching_order,
> >>
> >> random = ~ 1 | study / esid,
> >>
> >> sparse = TRUE)
> >>
> >>
> >> coef_test(resmod2, vcov = "CR2")
> >>
> >> #Using this simpler approaches, everything is estimated.
> >>
> >>
> >> *Question 1.*
> >>
> >> Could you confirm that the robust variance estimation is
> appropriate for
> >> our data (given that dependence between effect sizes are produced
> not only
> >> by the presence of several outcomes, but also by the presence of
> several
> >> independent variable measures)?
> >>
> >> *Question 2.*
> >>
> >> Is there an approach that should be absolutely privileged (we tend to
> >> believe that the CSE approach would be the most suitable) and is the
> >> implantation of the various models employing an appropriate syntax?
> >>
> >> *Question 3.*
> >>
> >> Is it correct to anticipate a within-study heterogeneity in true effect
> >> size according to the measure of outcome/independent variable while
> most of
> >> the studies (70%) used only one combination of outcome and independent
> >> variable measure?
> >>
> >>
> >> Best regards
> >>
> >> BF
> >>
> >>
> >> ######reproductible example
> >>
> >> library(clubSandwich)
> >>
> >>
> >> data(SATcoaching) # load data
> >>
> >>
> >> names(SATcoaching)[names(SATcoaching) == "d"] <- "z" # we used z as
> effect
> >> size
> >>
> >>
> >> SATcoaching$Vz <- 1 / ((SATcoaching$nT + SATcoaching$nC) - 3) #
> variance of
> >> our effect size
> >>
> >>
> >> names(SATcoaching)[names(SATcoaching) == "test"] <- "outcome"
> >>
> >>
> >>
> >> #here, we set that some studies have used a unique outcome (to let the
> >> measure of the independent variable vary)
> >>
> >>
> >> SATcoaching$outcome[12] <- "Math";
> >>
> >> SATcoaching$outcome[14] <- "Math";
> >>
> >> SATcoaching$outcome[17] <- "Math";
> >>
> >> SATcoaching$outcome[19] <- "Math";
> >>
> >> SATcoaching$outcome[33] <- "Math";
> >>
> >> SATcoaching$outcome[67] <- "Math";
> >>
> >>
> >>
> >> # we create a new variable which described that the independent
> variable
> >> has been measured using 3 different tools (A B or C)
> >>
> >>
> >> SATcoaching$measure_ind_variable <- "A";
> >>
> >> SATcoaching$measure_ind_variable[c(5,6)] <- c("B","C");
> >>
> >> SATcoaching$measure_ind_variable[c(12,13)] <- c("B","C");
> >>
> >> SATcoaching$measure_ind_variable[c(14,15)] <- c("B","C");
> >>
> >> SATcoaching$measure_ind_variable[c(16,17)] <- c("A","C");
> >>
> >> SATcoaching$measure_ind_variable[c(18,19)] <- c("A","C");
> >>
> >> SATcoaching$measure_ind_variable[c(32,33)] <- c("A","C");
> >>
> >> SATcoaching$measure_ind_variable[c(66,67)] <- c("A","C");
> >>
> >>
> >> # we reorder the dataframe
> >>
> >>
> >> SATcoaching_order <- SATcoaching[order(SATcoaching$study),]
> >>
> >>
> >>
> >> SATcoaching_order$esid <- 1:nrow(SATcoaching_order)
> >>
> >>
> >> #Then, we created a variable combining – for each study – the
> outcome and
> >> the measure of independent variable:
> >>
> >>
> >> SATcoaching_order$out_var <- with(SATcoaching_order,
> >>
> >> paste0(outcome, "_", measure_ind_variable))
> >>
> >> [[alternative HTML version deleted]]
> >>
> >>
> >>
> >>
> >> ------------------------------
> >>
> >> Subject: Digest Footer
> >>
> >> _______________________________________________
> >> R-sig-meta-analysis mailing list
> >> R-sig-meta-analysis using r-project.org
> >>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161157958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YgI%2BT5fhl925jleGSYW%2BgqSBoSW0bNrYUbvKMVw%2BT2A%3D&reserved=0
> >>
> >>
> >> ------------------------------
> >>
> >> End of R-sig-meta-analysis Digest, Vol 47, Issue 4
> >> **************************************************
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=04%7C01%7C%7C05e00cf1fc21434b57b608d91ed5dc14%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637574726161157958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YgI%2BT5fhl925jleGSYW%2BgqSBoSW0bNrYUbvKMVw%2BT2A%3D&reserved=0
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list