[R-meta] variance of predicted effect sizes

Fri Nov 16 16:26:35 CET 2018

Perfect and thanks again to all.

Best,

Hugh

Hugh F. Crean, Ph.D.
University of Rochester
School of Nursing
Helen Wood Hall 4.W111
601 Elmwood Avenue
Box SON
Rochester, New York 14642

(585) 276-5575
(585) 273-1258 (fax)

Hugh_Crean using urmc.rochester.edu

________________________________________
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> on behalf of r-sig-meta-analysis-request using r-project.org <r-sig-meta-analysis-request using r-project.org>
Sent: Friday, November 16, 2018 10:01 AM
To: r-sig-meta-analysis using r-project.org
Subject: R-sig-meta-analysis Digest, Vol 18, Issue 19

Send R-sig-meta-analysis mailing list submissions to
        r-sig-meta-analysis using r-project.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dmeta-2Danalysis&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=QS5zj5ym-9Jrivnk0ZgwN5MmQ9f6KoSud2NFnxffZhY&e=
or, via email, send a message with subject or body 'help' to
        r-sig-meta-analysis-request using r-project.org

You can reach the person managing the list at
        r-sig-meta-analysis-owner using r-project.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-meta-analysis digest..."

Today's Topics:

   1. Re: error installing devel version of metafor
      (Viechtbauer, Wolfgang (SP))
   2. Re:  Random-effects meta-analyses with inverse-variance
      weights cannot include studies with sample sizes of two?
      (Viechtbauer, Wolfgang (SP))
   3. Re: variance of predicted effect sizes
      (Viechtbauer, Wolfgang (SP))

----------------------------------------------------------------------

Message: 1
Date: Fri, 16 Nov 2018 14:08:42 +0000
From: "Viechtbauer, Wolfgang (SP)"
        <wolfgang.viechtbauer using maastrichtuniversity.nl>
To: "Gram, Gil (IITA)" <G.Gram using cgiar.org>,
        "r-sig-meta-analysis using r-project.org"
        <r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta] error installing devel version of metafor
Message-ID: <8d86d51d6e9e4e8ab20371cfc8ac97e8 using UM-MAIL3214.unimaas.nl>
Content-Type: text/plain; charset="utf-8"

I can't really help on the devtools issue.

As for the error you get when trying to run cooks.distance() (you don't have to call cooks.distance.rma.mv() explicitly -- the 'dispatch' from cooks.distance() to cooks.distance.rma.mv() happens automatically): Did you fit 'modRE_6' with an older version of metafor? Then try refitting it with the current version first.

If model fitting is slow and hence cooks.distance() is also very slow, you could use 'reestimate=TRUE'. That should speed things up a lot. See help(cooks.distance.rma.mv) and the 'Note' section.

Best,

Wolfgang

-----Original Message-----

From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Gram, Gil (IITA)

Sent: Thursday, 15 November, 2018 12:18

To: r-sig-meta-analysis using r-project.org

Subject: Re: [R-meta] error installing devel version of metafor

Hi Wolfgang,

It indeed seems to be not related to metafor, as it also doesn’t work for another package I tried. Have been trying to figure out the problem, but couldn’t find much (specific) help. I downloaded therefore the dev package metafor-master manually from https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_wviechtb_metafor&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=EVF5Ppgt-NKN5r7kzXj6NEM4m32TKbd6LEjEcLFvqDI&e= , and installed/loaded it as shown below. Two questions:

(1) I could not run cooks.distance.rma.mv() because of a new error (see below)

(2) profile.rma.mv() had a similar problem as cd with the 2.0-0 version, i.e. it was running for days without showing signs of progress. You think this should also go faster with the dev version 2.1-0?

install.packages("devtools")

library('devtools')

install('/Library/Frameworks/R.framework/Versions/3.5/Resources/library/metafor-master')

library('metafor’)

> packageVersion("metafor")

[1] '2.1.0'

> class(modRE_6)

[1] "rma.mv" “rma”

> summary(modRE_6)

Multivariate Meta-Analysis Model (k = 5130; method: REML)

…

coefficients 1:30

…

> cooks.distance.rma.mv(modRE_6, progbar=T)

Error in seq_len(x$k.all) :

  argument must be coercible to non-negative integer

In addition: Warning message:

In seq_len(x$k.all) : first element used of 'length.out' argument

Your help is appreciated.

Gil Gram

PhD researcher  |  Natural resource management/CCAFS

International Institute of Tropical Agriculture (IITA)

East African Hub/Kampala/Country office

Address: IITA-Uganda Plot 15B, Naguru East Road. P.O Box 7878, Kampala

Mobile: +256 755 315236  |  Belgium Mobile: +32 484 981200

Skype: gil.gram

On 14 Nov 2018, at 18:53, Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>> wrote:

Dear Gil,

This doesn't look like it is related to metafor per se. Can you install other packages via install_github()? Have you tried updating devtools (in case it is not the newest version)?

Best,

Wolfgang

-----Original Message-----

From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Gram, Gil (IITA)

Sent: Tuesday, 13 November, 2018 8:43

To: r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>

Subject: [R-meta] error installing devel version of metafor

Dear community and Wolfgang,

I’m trying to compute the cook’s distances for my pretty complex rma.mv model (for context, running the model with sparse=TRUE still takes about 12 hours). However, after a few days still at 0% I assume it’s not going well. I read that cooks.distance.rma.mv is indeed slow on complex models (https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_pipermail_r-2Dsig-2Dmeta-2Danalysis_2017-2DAugust_000122.html&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=NhUi2r4kQfaLP-xNkPrMbaCqaQdV08HBDhPXd0AaCcY&e= ) and Wolfgang suggested to install the devel version of metafor with install.packages("devtools") and devtools::install_github("wviechtb/metafor”). Unfortunately I get the following error: “> devtools::install_github("wviechtb/metafor")

Downloading GitHub repo wviechtb/metafor using master tar: Failed to set default locale tar: Failed to set default locale During startup - Warning messages: 1: Setting LC_CTYPE failed, using "C" 2: Setting LC_TIME failed, using "C" 3: Setting LC_MESSAGES failed, using "C" 4: Setting LC_MONETARY failed, using "C" v checking for file '/private/var/folders/wz/t5wd352s5wl7tz7crj5r3ddr0000gn/T/Rtmp8x3jdd/remotesfda76b9f309/wviechtb-metafor-e153805/DESCRIPTION' ... - preparing 'metafor': v checking DESCRIPTION meta-information ... - checking for LF line-endings in source and make files and shell scripts - checking for empty or unneeded directories - looking to see if a 'data/datalist' file should be added - building 'metafor_2.1-0.tar.gz'

Error: (converted from warning) Setting LC_CTYPE failed, using "C"

Execution halted Error in i.p(...) : (converted from warning) installation of package '/var/folders/wz/t5wd352s5wl7tz7crj5r3ddr0000gn/T//Rtmp8x3jdd/filefda18e204cb/metafor_2.1-0.tar.gz' had non-zero exit status”

Tried to look for other ways of installing it, but in vain. Hope someone can help with this? Not sure what other information I should provide...

thanks!

Gil

------------------------------

Message: 2
Date: Fri, 16 Nov 2018 14:51:30 +0000
From: "Viechtbauer, Wolfgang (SP)"
        <wolfgang.viechtbauer using maastrichtuniversity.nl>
To: Patrizio Tressoldi <patrizio.tressoldi using unipd.it>,
        "r-sig-meta-analysis using r-project.org"
        <r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta]  Random-effects meta-analyses with
        inverse-variance weights cannot include studies with sample sizes of
        two?
Message-ID: <f710214f80014e9187fdcec8fa429aad using UM-MAIL3214.unimaas.nl>
Content-Type: text/plain; charset="utf-8"

I can't give any specific advice here. This would require knowing more details of the studies, the data collection procedures, and the goals of the meta-analysis. I believe this article is quite relevant though:

Bakbergenuly, I., Kulinskaya, E., & Morgenthaler, S. (2016). Inference for binomial probability based on dependent Bernoulli random variables with applications to meta-analysis and group level studies. Biometrical Journal, 58(4), 896-914. doi:10.1002/bimj.201500115

Best,

Wolfgang

-----Original Message-----

From: Patrizio Tressoldi [mailto:patrizio.tressoldi using unipd.it]

Sent: Friday, 09 November, 2018 16:20

To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org

Subject: Re: [R-meta] Random-effects meta-analyses with inverse-variance weights cannot include studies with sample sizes of two?

Thank you very much Wolfgang for your exaustive information.

X is based on the performance of a single participant.

Given you raised doubts about the independence assumption, what do you

suggest as a best effect size and variance alternative in similar cases?

Best

Patrizio

Il 09/11/2018 14:33, Viechtbauer, Wolfgang (SP) ha scritto:

> I see. This assumes independence between trials and is typically used when measuring a dichotomous outcome on N different subjects. But here it seems like a single subject went through N different trials. In that case, the independence assumption seems tenuous.

>

> How does this relate to the two subjects you talked about earlier? Or is X based on the sum of two subjects?

>

> Also, I don't see how you then derived the equation for the variance of z/sqrt(N). You wrote:

>

> Variance = Sqr(1/(1-q)*(1-1/(1-q))/N

>

> But let's go through this step-by-step:

>

> Leaving aside the independence issue, we assume (for large enough N):

>

> p ~ N(π, π(1-π)/N)

>

> Then we compute:

>

> z = (p - π₀) / sqrt(π₀(1-π₀)/N)

>

> Under H0: π = π₀, it is easy to see that z ~ N(0,1), so z/sqrt(N) ~ N(0, 1/N).

>

> But if H0 is not true, then some simple algebra shows:

>

> Var(z) = π(1-π) / (π₀(1-π₀))

>

> and hence Var(z/sqrt(N)) = π(1-π) / (N * π₀(1-π₀)), which we could estimate with:

>

> Var(z/sqrt(N)) = p(1-p) / (N * π₀(1-π₀)).

>

> Best,

> Wolfgang

>

> -----Original Message-----

> From: Patrizio Tressoldi [mailto:patrizio.tressoldi using unipd.it]

> Sent: Friday, 09 November, 2018 13:01

> To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org

> Subject: Re: [R-meta] Random-effects meta-analyses with inverse-variance weights cannot include studies with sample sizes of two?

>

> Il 09/11/2018 12:05, Viechtbauer, Wolfgang (SP) ha scritto:

>

> Maybe I am a bit dense here, but I still do not fully understand what you are computing. You wrote earlier that there are two participants that "contributed to a percentage of hits, e.g. .30 and .40". Ok, that sounds like these participants did a series of trials that could yield hits/successes. I assume N is the number of trials. So the first participant had .3*N hits and the second participant had .4*N hits. So far so good. But what is 'z = binomial z'? Where does that equation for the variance come from? It would also help if you could provide a fully reproducible example of the computations.

>

> This is the z score obtained from a normal approximation of the binomial test binomial test: z= (X - µ)/σ ; where X = observed percentage;  µ = chance percentage; σ = Sqr(µ(1-µ)/N

>

> ex. X= 31; N= 49; p = .5 = (31/46 - .5)/Sqr(.5*.5/49) = 2.4

>

> Patrizio

>

> --

> Patrizio E. Tressoldi Ph.D.

> Dipartimento di Psicologia Generale

> Università di Padova

> via Venezia 8

> 35131 Padova - ITALY

> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.patriziotressoldi.it&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=Sd563itSh6i7OWsvtGPqu-SO6a1rYJRSkGfud52CU3Y&e=

> https://urldefense.proofpoint.com/v2/url?u=https-3A__orcid.org_0000-2D0002-2D6404-2D0058&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=C3CYeGPGFf4fvri2vDSOrpnf2JygAWNu3ebmHIWaGLg&e=

>

> Science of Consciousness Research Group

> https://urldefense.proofpoint.com/v2/url?u=http-3A__dpg.unipd.it_en_soc&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=Xfb_YHxBlV8XjLqqgdWVUBUWRS-z1EWYXEr4T9KXa9w&e=

>

> Make war history

> support https://urldefense.proofpoint.com/v2/url?u=http-3A__www.emergency.it&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=vkT5Ye0OT2JdlVjjOKhSqLMxI59LZBC2013wEH39pZE&e=

------------------------------

Message: 3
Date: Fri, 16 Nov 2018 14:55:59 +0000
From: "Viechtbauer, Wolfgang (SP)"
        <wolfgang.viechtbauer using maastrichtuniversity.nl>
To: "Crean, Hugh" <hugh_crean using URMC.Rochester.edu>
Cc: "r-sig-meta-analysis using r-project.org"
        <r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta] variance of predicted effect sizes
Message-ID: <f9138838f2574b07b991af3a78b2b9e5 using UM-MAIL3214.unimaas.nl>
Content-Type: text/plain; charset="us-ascii"

Please always cc the mailing list when replying.

In metafor, predict() gives standard errors of predicted values, labeled 'se' in the output and element 'se' in the object returned by predict().

Best,
Wolfgang

-----Original Message-----
From: Crean, Hugh [mailto:hugh_crean using URMC.Rochester.edu]
Sent: Friday, 16 November, 2018 15:52
To: Viechtbauer, Wolfgang (SP)
Subject: Re: variance of predicted effect sizes

Hi Wolfgang,

Yes, that is it exactly.  What I cannot find from your response below is "...Then you would use the squared standard error of the predicted value (from the meta-regression model) as the sampling variance of the control group estimate...."  I cannot find (or figure a sensible formula for computign) the standard error of the predicted value.

Best,

Hugh

Hugh F. Crean, Ph.D.
University of Rochester
School of Nursing
Helen Wood Hall 4.W111
601 Elmwood Avenue
Box SON
Rochester, New York 14642

(585) 276-5575
(585) 273-1258 (fax)

Hugh_Crean using urmc.rochester.edu

________________________________________
From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
Sent: Friday, November 16, 2018 9:46 AM
To: Crean, Hugh
Subject: RE: variance of predicted effect sizes

Dear Hugh,

It sounds to me that you want to do something like Becker (1988) describes in section 5.2. Then you would use the squared standard error of the predicted value (from the meta-regression model) as the sampling variance of the control group estimate of the standardized mean change.

There is an additional complication that there is then a dependency between the effect sizes (i.e., the difference in the standardized mean change between the treatment and control group) for studies with both treatment and control groups and effect sizes for studies with just a treatment group (and also between the effect sizes from studies with treatment groups only). These covariances can be computed as described in 5.2 and would need to be put into the 'V' matrix of rma.mv() (if you intend to use metafor). Actually implemting this would require a bit of work though. Showing how to do this with metafor would be a nice little exercise/project for a motivated student.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Crean, Hugh
Sent: Sunday, 11 November, 2018 23:22
To: 'r-sig-meta-analysis using r-project.org'
Subject: [R-meta] variance of predicted effect sizes

Hello,

Colleagues and myself are working on a meta-analysis of sleep interventions.  Many of the studies are only single arm pre-post studies and we are following the advice of Becker (1988) and Morris and DeShon (2002) to impute missing control group effect sizes.  We are planning on using meta regression to compute predicted effect sizes for those studies missing control information.  However, I cannot quite figure how to compute and/or get the standard error for this estimate.  Would one run a simple meta- analysis on the predicted scores for those with the data and use the provided se (and variance)?

Thanks in advance,

Hugh

Hugh F. Crean, Ph.D.
School of Nursing
University of Rochester
601 Elmwood Avenue
Rochester, New York  14620

(585) 276-5575

------------------------------

Subject: Digest Footer

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis using r-project.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dsig-2Dmeta-2Danalysis&d=DwIGaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=y1u-he5yH9LDOYtsYlCaO4_gbrlnAM_rtoLiKSa3yyI&m=fPjx-W2LskrB0rOJK7nS0CDcMJ04jBoTYFMLgKFfVZw&s=QS5zj5ym-9Jrivnk0ZgwN5MmQ9f6KoSud2NFnxffZhY&e=

------------------------------

End of R-sig-meta-analysis Digest, Vol 18, Issue 19
***************************************************