[R-meta] Unweighted analysis, Re: [EXT] R-sig-meta-analysis Digest, Vol 82, Issue 31
Alan Wilson
w||@on @end|ng |rom @uburn@edu
Tue Mar 26 14:07:46 CET 2024
Thanks, Michael. Some of the effect sizes that my students are working with are not standard and don't have error. I don't think imputing error for effect sizes when no error is available is a good idea, which is why I was trying to figure out how to use metafor for their analyses. I guess we will just use standard parametric statistics to calculate mean effects and 95% CI. Thanks again for your time. alan
-----Original Message-----
From: Michael Dewey <lists using dewey.myzen.co.uk>
Sent: Tuesday, March 26, 2024 8:02 AM
To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
Cc: Alan Wilson <wilson using auburn.edu>
Subject: Unweighted analysis, Re: [R-meta] [EXT] R-sig-meta-analysis Digest, Vol 82, Issue 31
Dear Alan
If your students do not know how variable the primary studies are then they are not going to be able to tell how variables the summary is.
Are they sure they have exhausted all the ways of back-calculating the standard errors from other statistics? It would be quite unusual for authors not to give a signficance test of some form. They could also try imputing the standard errors from other studies especially if some of the studies at hand do have standard errors.
Michael
On 25/03/2024 18:16, Alan Wilson via R-sig-meta-analysis wrote:
> Thanks for the advice, Michael. Typically, my students who are
> running unweighted analyses have no variance estimates for their
> effect sizes. Since vi is required, we need to put something into the
> vi column. Seeing the variation in 95% CIs around the mean effect
> depending on what vi values I used was concerning. Does anyone have
> advice for the best practice for unweighted analyses when vi values
> are not available? Thanks in advance. alan
>
> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org>
> On Behalf Of r-sig-meta-analysis-request using r-project.org
> Sent: Thursday, March 21, 2024 11:19 AM
> To: r-sig-meta-analysis using r-project.org
> Subject: [EXT] R-sig-meta-analysis Digest, Vol 82, Issue 31
>
> CAUTION: Email Originated Outside of Auburn.
>
> Send R-sig-meta-analysis mailing list submissions to
> r-sig-meta-analysis using r-project.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://stat/
> .ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7Cwil
> son%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Cccb6deedbd294b38
> 8979d72780f62d3b%7C0%7C0%7C638470549845910049%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0
> %7C%7C%7C&sdata=vdZri81iNIDnCXfDxiA5xFkNcz2NJpXpcMjkZDEo%2Bw8%3D&reser
> ved=0 or, via email, send a message with subject or body 'help' to
> r-sig-meta-analysis-request using r-project.org
>
> You can reach the person managing the list at
> r-sig-meta-analysis-owner using r-project.org
>
> When replying, please edit your Subject line so it is more specific than "Re: Contents of R-sig-meta-analysis digest..."
>
>
> Today's Topics:
>
> 1. unweighted analysis questions (Alan Wilson)
> 2. Re: unweighted analysis questions (Michael Dewey)
> 3. Re: Prediction Intervals for estimates of random slope
> rma.mv model with predict.rma() (Zac Robinson)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 21 Mar 2024 14:51:06 +0000
> From: Alan Wilson <wilson using auburn.edu>
> To: "r-sig-meta-analysis using r-project.org"
> <r-sig-meta-analysis using r-project.org>
> Subject: [R-meta] unweighted analysis questions
> Message-ID:
>
> <MW2PR1901MB4697CAA9F67A1C8E78FCBEE9BF322 using MW2PR1901MB4697.namprd19.pro
> d.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hey everyone - I have two questions regarding unweighted analyses.
>
> 1) Using the suggested code for unweighted analysis (rma(yi, vi, method="FE", data=dat, weighted=FALSE)) provides the same main effect but different 95% CIs when using the same or different vi values across studies. I also found different 95% CIs when testing two different vi values (1 vs 100) that were the same across studies. I also found that changing weighted=FALSE to weighted=TRUE provided same 95% CIs when using the same dataset. Any idea what is causing the variation in 95% CIs? What is best for unweighted analyses?
>
> 2) Any suggested approaches for publication bias checks for unweighted analyses? I was thinking funnel plots of effect size and sample size.
>
> Thanks for any advice. alan
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 21 Mar 2024 15:27:00 +0000
> From: Michael Dewey <lists using dewey.myzen.co.uk>
> To: R Special Interest Group for Meta-Analysis
> <r-sig-meta-analysis using r-project.org>
> Cc: Alan Wilson <wilson using auburn.edu>
> Subject: Re: [R-meta] unweighted analysis questions
> Message-ID: <b5c372ec-dbde-1825-b980-1974e04e70c5 using dewey.myzen.co.uk>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>
> Dear Alan
>
> On 21/03/2024 14:51, Alan Wilson via R-sig-meta-analysis wrote:
>> Hey everyone - I have two questions regarding unweighted analyses.
>>
>> 1) Using the suggested code for unweighted analysis (rma(yi, vi, method="FE", data=dat, weighted=FALSE)) provides the same main effect but different 95% CIs when using the same or different vi values across studies. I also found different 95% CIs when testing two different vi values (1 vs 100) that were the same across studies. I also found that changing weighted=FALSE to weighted=TRUE provided same 95% CIs when using the same dataset. Any idea what is causing the variation in 95% CIs? What is best for unweighted analyses?
>>
>
> That is what I would have expected. Although the vi are not being used as weights they do still tell you about the variability. If you combine lots of very imprecise studies you would not expect to get the same overall CI as combining lots of very precise studies. Otherwise you would be getting a free lunch.
>
>> 2) Any suggested approaches for publication bias checks for unweighted analyses? I was thinking funnel plots of effect size and sample size.
>>
>
> Other people will probably pitch in with evidence-based ideas here but I would go for sqrt sample size I think.
>
>
> Michael
>
>> Thanks for any advice. alan
>>
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://sta/
>> t.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7Cw
>> ilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Cccb6deedbd294
>> b388979d72780f62d3b%7C0%7C0%7C638470549845918376%7CUnknown%7CTWFpbGZs
>> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>> D%7C0%7C%7C%7C&sdata=oZnNy090t1HiivVdh%2BXj2gAz8PwjQNL40I7tYKHWzAM%3D
>> &reserved=0
>>
>
> --
> Michael
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 21 Mar 2024 12:18:41 -0400
> From: Zac Robinson <zacrobinson2015 using gmail.com>
> To: R Special Interest Group for Meta-Analysis
> <r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Prediction Intervals for estimates of random
> slope rma.mv model with predict.rma()
> Message-ID:
>
> <CALM2kOaeKnJ4z-Kwbu9uTyN8eU=rZuK87iD5aNo5_agi47N-fA using mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Wolfgang,
>
> Apologies for the incorrect method to reply - hopefully got it right
> this time. If I'm understanding things correctly - the coding can
> (potentially) end up giving me what I'm after, but I'm just specifying
> the sigma / sd for the prediction interval incorrectly (i.e., the
> variances can't just be summed).
>
> Thus, I guess my follow up question is - how would you recommend going
> about this calculation? The closest thing I can find in the literature
> with a similar variance calculation is in the context of R2 from
> Johnson et al., where they calculate the "mean random effects
> variance" which seems like it could be useful here:
>
> https://besj/
> ournals.onlinelibrary.wiley.com%2Fdoi%2F10.1111%2F2041-210X.12225&data
> =05%7C02%7Cwilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Ccc
> b6deedbd294b388979d72780f62d3b%7C0%7C0%7C638470549845923786%7CUnknown%
> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> VCI6Mn0%3D%7C0%7C%7C%7C&sdata=y8aJo27gBdkFH0ZGhz%2Fpn3IjSh0T7icVj%2F6f
> EAOQBiE%3D&reserved=0
>
> Let me know what you think and thank you for your time and expertise,
>
> Zac
>
> On Thu, Mar 14, 2024 at 9:23 AM Viechtbauer, Wolfgang (NP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
>> Dear Zac,
>>
>> Please respond to the mailing list, not just to me personally.
>>
>> Just summing up all the variances isn't correct - the correct
>> computation of the prediction interval for such a random-effects
>> structure is unfortunately more complex.
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: Zac Robinson <zacrobinson2015 using gmail.com>
>>> Sent: Monday, March 11, 2024 16:07
>>> To: Viechtbauer, Wolfgang (NP) <
>> wolfgang.viechtbauer using maastrichtuniversity.nl>
>>> Subject: Re: [R-meta] Prediction Intervals for estimates of random
>>> slope
>> rma.mv
>>> model with predict.rma()
>>>
>>> Hey Wolfgang,
>>>
>>> Thank you very much for the reply. I actually have played around
>>> with
>> things a
>>> bit more and think I've found a solution. I may not be specifying
>>> the
>> sigma
>>> argument perfectly for a random slope model - but I think in concept
>> this works!
>>> Looks like emmeans / predict() have cross compatibility that allows
>>> for prediction intervals when sigma is specified
>>>
>>> Let me know what you think,
>>> Zac
>>>
>>> # SETUP
>> -------------------------------------------------------------------
>>> #load packages and data
>>> library(metafor)
>>> library(tidyverse)
>>> library(emmeans)
>>>
>>> data("dat.konstantopoulos2011")
>>> df=dat.konstantopoulos2011%>%
>>> as.data.frame()%>%
>>> mutate(factor=sample(c("A","B"),56,replace=TRUE))
>>>
>>> # RANDOM SLOPE (x1)
>> -------------------------------------------------------
>>> #fit single random slope model
>>> m2= rma.mv(yi ~ year+factor, vi,
>>> random = list(~year|study, ~1|district, ~1|school),
>>> data = df, method = "REML",struct = "GEN",
>>> control = list(optimizer="bobyqa"))
>>>
>>> #extract estimates with emmmprep + emmeans + predict
>>> est.ci=emmprep(m2,
>>> at=list(year=1976:2000))%>%
>>> emmeans(~year,
>>> weights = "prop",
>>> sigma=sqrt(sum(
>>> m2$sigma2,
>>> m2$tau2,
>>> m2$gamma2
>>> )))
>>>
>>> pi= est.ci%>%
>>> predict(interval="prediction")%>%
>>> select(lower.PL,upper.PL)
>>>
>>> preds=cbind(est.ci,pi)
>>> preds # prediction intervals are now added
>>>
>>> ggplot(data = preds,
>>> aes(x=year,
>>> y=emmean))+theme_classic()+
>>> geom_point(data=df,
>>> aes(y=yi,
>>> size=1/vi))+
>>> geom_ribbon(aes(ymin=lower.PL,
>>> ymax=upper.PL),
>>> linetype="dashed",
>>> alpha=0,
>>> color="black")+
>>> geom_ribbon(aes(ymin=asymp.LCL,
>>> ymax=asymp.UCL),
>>> alpha=0.25)+
>>> geom_line()
>>>
>>> On Mon, Mar 11, 2024 at 2:06 AM Viechtbauer, Wolfgang (NP)
>>> <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>> Dear Zac,
>>>
>>> predict() currently does not provide predicition intervals for
>> http://rma/.
>> mv%2F&data=05%7C02%7Cwilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d
>> 950e2b%7Cccb6deedbd294b388979d72780f62d3b%7C0%7C0%7C63847054984592913
>> 3%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
>> 6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Pm3e0ZH72zJHoktoCm9GKnn7Ck
>> EZBMWMP3y2OXFXCrw%3D&reserved=0
>>> models with a GEN random effects structure. Not sure when I will get
>> around to
>>> implementing that.
>>>
>>> Best,
>>> Wolfgang
>>>
>>>> -----Original Message-----
>>>> From: R-sig-meta-analysis <mailto:
>> r-sig-meta-analysis-bounces using r-project.org>
>>> On Behalf
>>>> Of Zac Robinson via R-sig-meta-analysis
>>>> Sent: Sunday, March 10, 2024 16:26
>>>> To: mailto:r-sig-meta-analysis using r-project.org
>>>> Cc: Zac Robinson <mailto:zacrobinson2015 using gmail.com>
>>>> Subject: [R-meta] Prediction Intervals for estimates of random
>>>> slope
>>> http://rma/
>>> .mv%2F&data=05%7C02%7Cwilson%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc
>>> 4d950e2b%7Cccb6deedbd294b388979d72780f62d3b%7C0%7C0%7C63847054984593
>>> 3935%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
>>> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=9O97NGQj6cbfYQLmYFLPg
>>> TBhapqpa3BePe9LnzhIgBs%3D&reserved=0
>>>> model with predict.rma()
>>>>
>>>> Dear All,
>>>>
>>>> I am working on extracting the adjusted effects from a
>>>> dose-response multilevel meta-regression. One issue I've run into
>>>> is getting the appropriate prediction intervals on my estimates
>>>> when the model includes random slopes (specifically when struct = "GEN" is included).
>> From
>>>> what I gather from the help files, this comes down to appropriately
>>>> specifying the `tau2.levels` and `gamma2.levels` arguments within
>>>> predict.rma() - however, I am unsure how to get this to work.
>>>>
>>>> In the code below I've tried to demonstrate my issue and explore
>>>> the
>> other
>>>> solutions I'm aware of through the orchaRd package). My end goal
>> requires
>>>> adjusting for both continuous and categorical moderators so ideally
>>>> the solution stays within the emmprep() / emmeans() framework
>>>> (which makes proportional weights and setting non-focal continuous
>>>> variables at the
>> mean
>>>> easy), but I realize this may not be possible.
>>>>
>>>> Thank you very much in advance!
>>>>
>>>> Zac
>>>>
>>>> # SETUP
>> -------------------------------------------------------------------
>>>> #load packages and data
>>>> library(metafor)
>>>> library(tidyverse)
>>>> library(orchaRd)
>>>> library(emmeans)
>>>>
>>>> data("dat.konstantopoulos2011")
>>>> df=dat.konstantopoulos2011%>%
>>>> as.data.frame()%>%
>>>> mutate(factor=sample(c("A","B"),nrow(df),replace=TRUE))
>>>>
>>>> # RANDOM INTERCEPT
>> --------------------------------------------------------
>>>> #fit random intercept model
>>>> m1=http://rma.mv/(yi ~ year+factor, vi,
>>>> random = list(~1|study, ~1|district, ~1|school),
>>>> data = df, method = "REML")
>>>>
>>>> #extract estimates with predict.rma predict(m1,
>>>> newmods =cbind(rep(c(1976:2000),2),
>>>> rep(c(0,1),each=25)),
>>>> addx = TRUE) # works fine
>>>>
>>>> #extract estimates with emmprep
>>>> emmprep(m1,
>>>> at=list(year=1976:2000))%>%
>>>> emmeans(~year,
>>>> weights = "prop")%>%
>>>> pred_interval_esmeans(model=m1,
>>>> mm=.,
>>>> mod="year") # works fine
>>>>
>>>> #extract estimates with orchaRd
>>>> mod_results(m1,
>>>> mod = "year",
>>>> group = "study") # works fine
>>>>
>>>> # RANDOM SLOPE (x1)
>> -------------------------------------------------------
>>>> #fit single random slope model
>>>> m2=http://rma.mv/(yi ~ year+factor, vi,
>>>> random = list(~year|study, ~1|district, ~1|school),
>>>> data = df, method = "REML",struct = "GEN",
>>>> control = list(optimizer="bobyqa"))
>>>>
>>>> #extract estimates with predict.rma predict(m2, newmods
>>>> =cbind(rep(c(1976:2000),2),
>>>> rep(c(0,1),each=25)),
>>>> addx = TRUE) # no prediction intervals
>>>>
>>>> predict(m2, newmods =cbind(rep(c(1976:2000),2),
>>>> rep(c(0,1),each=25)),
>>>> addx = TRUE,
>>>> tau2.levels = levels(year), # try adding tau2.levels
>>>> ) # still no prediction intervals
>>>>
>>>> #extract estimates with emmprep
>>>> emmprep(m2,
>>>> at=list(year=1976:2000))%>%
>>>> emmeans(~year,
>>>> weights = "prop")%>%
>>>> pred_interval_esmeans(model=m2,
>>>> mm=.,
>>>> mod="year") # works but gives an error
>>>>
>>>> #extract estimates with orchaRd
>>>> mod_results(m2,
>>>> mod = "year",
>>>> group = "study") # works but gives an error
>>>>
>>>>
>>>> # RANDOM SLOPE (x2)
>> -------------------------------------------------------
>>>> #fit double random slope model
>>>> m3=http://rma.mv/(yi ~ year+factor, vi,
>>>> random = list(~year|study, ~year|district, ~1|school),
>>>> data = df, method = "REML",struct = c("GEN","GEN"),
>>>> control = list(optimizer="bobyqa"))
>>>>
>>>> #extract estimates with predict.rma predict(m3, newmods
>>>> =cbind(rep(c(1976:2000),2),
>>>> rep(c(0,1),each=25)),
>>>> addx = TRUE) # no prediction intervals
>>>>
>>>> predict(m3, newmods =cbind(rep(c(1976:2000),2),
>>>> rep(c(0,1),each=25)),
>>>> addx = TRUE,
>>>> tau2.levels = levels(year),# try adding tau2.levels
>>>> gamma2.levels = levels(year)# try adding gamma2.levels
>>>> ) # still no prediction intervals
>>>>
>>>> #extract estimates with emmprep
>>>> emmprep(m3,
>>>> at=list(year=1976:2000))%>%
>>>> emmeans(~year,
>>>> weights = "prop")%>%
>>>> pred_interval_esmeans(model=m3,
>>>> mm=.,
>>>> mod="year") # works but gives an error
>>>>
>>>> #extract estimates with orchaRd
>>>> mod_results(m3,
>>>> mod = "year",
>>>> group = "study") # works but gives an error
>>
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat/
> .ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7Cwil
> son%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Cccb6deedbd294b38
> 8979d72780f62d3b%7C0%7C0%7C638470549845954353%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0
> %7C%7C%7C&sdata=f3EMtIeGZgpzsjktZrJzJuXjEjYUYO9%2FdbKbfO08XzI%3D&reser
> ved=0
>
>
> ------------------------------
>
> End of R-sig-meta-analysis Digest, Vol 82, Issue 31
> ***************************************************
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat/
> .ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7Cwil
> son%40auburn.edu%7C2d94e8b5f80d4aa6122f08dc4d950e2b%7Cccb6deedbd294b38
> 8979d72780f62d3b%7C0%7C0%7C638470549845959483%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0
> %7C%7C%7C&sdata=4qWG7jHmswU8qI0gALxrOaHw1vSMD8VozdITnySdyJg%3D&reserve
> d=0
>
>
--
Michael
More information about the R-sig-meta-analysis
mailing list