[R-meta] specification of phi in vcalc
James Pustejovsky
jepu@to @end|ng |rom gm@||@com
Mon Aug 15 20:58:16 CEST 2022
Hi Danielle,
My reaction is very similar to Michael's. These results don't seem
especially sensitive to the choice of phi. I think you might be able to
simply report them as is and note that the significance of one of the
coefficients is somewhat sensitive to the assumed value of
auto-correlation.
The only other thing that occurs to me is that you might also consider
reporting sensitivity analyses based on models with other random effect
structures, such as struct = "HCS" or struct = "CS" or struct = "ID". With
the model that you've fit, struct = "CAR", it seems that there is little or
no correlation between the random effects for a given study at different
time points. As a result, the estimated average effects at each time point
are more or less equivalent to what you would get from estimating separate
random effects models for the effect sizes from a given time-point. Using
other struct arguments might give some indication of the robustness of this
pattern.
James
On Mon, Aug 15, 2022 at 7:57 AM Michael Dewey <lists using dewey.myzen.co.uk>
wrote:
> Dear Danielle
>
> I do not pretend to be an expert on these more complicated models but to
> my eyes the output from all values of phi is very similar. Are you
> perhaps over-interpreting a small change in the p-value which takes it
> from one side of an arbitrary cut-off to another?
>
> Michael
>
> On 14/08/2022 01:21, Danielle Hiam wrote:
> > Hi James,
> >
> > Very happy to share the data. To give you a brief overview, I am trying
> to run a correlated and hierarchical meta-analysis to investigate changes
> in microRNA expression after a acute bout of exercise. Each of the primary
> studies have differing time points so I have picked the most “common”
> timepoints (PRE, POST, 1-2hour post exercise and 24 hours post exercise)
> and dummy coded them as follows -1 (PRE), 0(POST), 1(within 1-2HP) and 2
> (24HP). I have chosen to use the fold change from PRE as values as these
> would then be standard measurement across all the studies.
> >
> > * How many studies you've got:
> > I am doing multiple meta-analysis of several microRNAs. To give you an
> idea miRNA-1 has 14 studies, miR-133a 16 studies etc...
> >
> > * How many effect sizes per study (on average and the range of effect
> sizes per study; quartiles would be great too): miRNA-1: mean: 2.30 (total
> 46obs). miR-133a= mean 2.22 (total 51obs)
> > Average, median and IQR for miR-133a
> > Time mean MEDIAN IQR
> > 1 -1 1 1 0
> > 2 0 2.23 1.2 1.71
> > 3 1 1.54 1.49 1.08
> > 4 2 1.42 1.27 1.06
> >
> > * Your model specification, including both what moderators are included
> and how you've specified the random effects:
> > This is my code:
> > V <- vcalc(FC_SD, cluster=cohort.in.study,
> > time1=Time_NUM, data = dat2, phi=0.6)
> >
> > rma.mv(yi = FC_MEAN,
> > V = V,
> > data = dat2,
> > mods = ~ factor(Time_NUM),
> > random = list (~ Time_NUM|cohort.in.study),
> > struct = "CAR")
> > As the timepoints are not evenly spaced but are dependent (repeated
> measures) I ran CAR for the struct argument.
> > Cohort.in.study is a unique ID to indicate that the timepoints within
> the cohort are dependent while timepoints outside this cohort are
> independent.
> >
> > * For any included moderators, whether they are study-level
> characteristics (constant across the effect sizes for a given study) or
> effect-level characteristics (that vary within study).
> > I have chosen timepoints that are common across the studies (PRE, POST,
> 1-2hour post exercise and 24 hours post exercise). For example I dummy
> coded them as follows -1 (PRE), 0(POST), 1(within 1-2HP) and 2 (24HP)
> >
> > * If you're open to sharing, the results of fitting the model with
> different values of phi, to provide a sense of how much the estimates
> change.
> > No worries see below for the miRNA-133a results. You will see while
> estimates don’t change drastically, the results from phi 0.4, 0.5 and 0.6
> are borderline significant and when I change the phi to be above 0.7 they
> become significant. The QE also changes but it remains significant for all
> values of phi.
> >
> > Phi 0.4 estim sqrt fixed
> > tau^2 1.791 1.338 no
> > rho 0.000 no
> > Test for Residual Heterogeneity:
> > QE(df = 47) = 192.076, p-val < .001
> > Model Results:
> > estimate se¹ tval¹ df¹
> pval¹ ci.lb¹ ci.ub¹ <U+200B>
> > intrcpt 0.983 0.040 24.415 12.72
> <.001 0.896 1.070 ***
> > factor(Time_NUM)0 1.353 0.571 2.368 16.51
> 0.030 0.145 2.560 *
> > factor(Time_NUM)1 0.773 0.226 3.416 11.63
> 0.005 0.278 1.268 **
> > factor(Time_NUM)2 0.519 0.242 2.146 7.06
> 0.069 -0.052 1.089 .
> >
> >
> > Phi 0.5 estim sqrt fixed
> > tau^2 1.824 1.351 no
> > rho 0.000 no
> > Test for Residual Heterogeneity:
> > QE(df = 47) = 213.811, p<0.001
> > Model Results:
> > estimate se¹ tval¹ df¹
> pval¹ ci.lb¹ ci.ub¹ <U+200B>
> > intrcpt 0.978 0.050 19.485 12.76
> <.001 0.870 1.087 ***
> > factor(Time_NUM)0 1.361 0.580 2.348 16.45
> 0.032 0.135 2.587 *
> > factor(Time_NUM)1 0.800 0.235 3.409 11.56
> 0.005 0.286 1.313 **
> > factor(Time_NUM)2 0.541 0.240 2.257 7.05
> 0.058 -0.025 1.106 .
> >
> > Phi 0.6 estim sqrt fixed
> > tau^2 1.865 1.366 no
> > rho 0.000 no
> > Test for Residual Heterogeneity:
> > QE(df = 47) = 249.228, p<0.001
> > Model Results:
> > estimate se¹ tval¹ df¹
> pval¹ ci.lb¹ ci.ub¹ <U+200B>
> > intrcpt 0.974 0.060 16.213 12.82
> <.001 0.844 1.104 ***
> > factor(Time_NUM)0 1.369 0.588 2.328 16.39
> 0.033 0.125 2.614 *
> > factor(Time_NUM)1 0.822 0.244 3.376 11.48
> 0.006 0.289 1.356 **
> > factor(Time_NUM)2 0.564 0.239 2.358 7.01
> 0.050 -0.001 1.130 .
> >
> > Phi 0.8 estim sqrt fixed
> > tau^2 1.965 1.402 no
> > rho 0.000 no
> > Test for Residual Heterogeneity:
> > QE(df = 47) = 439.477, p<0.001
> > Model Results:
> > estimate se¹ tval¹ df¹
> pval¹ ci.lb¹ ci.ub¹ <U+200B>
> > intrcpt 0.965 0.079 12.157 12.95
> <.001 0.794 1.137 ***
> > factor(Time_NUM)0 1.387 0.606 2.291 16.27
> 0.036 0.105 2.670 *
> > factor(Time_NUM)1 0.857 0.262 3.265 11.24
> 0.007 0.281 1.433 **
> > factor(Time_NUM)2 0.612 0.246 2.493 6.90
> 0.042 0.030 1.194 *
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org>
> On Behalf Of James Pustejovsky
> > Sent: Thursday, 4 August 2022 1:12 PM
> > To: Lukasz Stasielowicz <lukasz.stasielowicz using uni-osnabrueck.de>
> > Cc: R meta <r-sig-meta-analysis using r-project.org>
> > Subject: Re: [R-meta] specification of phi in vcalc
> >
> > Hi Danielle,
> >
> > Just to add a little to Lukasz's suggestions (which I think are
> excellent), it is a little unusual and surprising that your results are
> very sensitive to changing the value of phi from 0.6 to 0.8. If you can
> provide a bit more detail about the structure of your data and model, I may
> be able to offer some suggestions on how to interpret this sensitivity.
> Specifically, it would be useful to know:
> > * How many studies you've got
> > * How many effect sizes per study (on average and the range of effect
> sizes per study; quartiles would be great too)
> > * Your model specification, including both what moderators are included
> and how you've specified the random effects
> > * For any included moderators, whether they are study-level
> characteristics (constant across the effect sizes for a given study) or
> effect-level characteristics (that vary within study).
> > * If you're open to sharing, the results of fitting the model with
> different values of phi, to provide a sense of how much the estimates
> change.
> >
> > Incidentally, this issue is closely related to something I'm studying
> right
> > now: https://www.jepusto.com/talk/srsm-2022-matter-of-emphasis/
> >
> > James
> >
> > On Wed, Aug 3, 2022 at 10:14 AM Lukasz Stasielowicz <
> lukasz.stasielowicz using uni-osnabrueck.de> wrote:
> >
> >> Dear Danielle,
> >>
> >> vcalc documentation contains a potentially helpful tip: "Argument phi
> >> must then also be specified to indicate the autocorrelation among the
> >> sampling errors of two effect sizes that differ by one unit on the
> >> time1 variable. As above, the autocorrelation of the measurements
> >> themselves can be used here as a proxy."
> >> https://rdrr.io/github/wviechtb/metafor/man/vcalc.html
> >>
> >> Perhaps some tables in primary studies show correlations between
> >> neighboring time points?
> >>
> >> Another option: If raw data are available for some primary studies,
> >> then one could estimate the correlation for several data sets and use
> >> it as "phi" input for vcalc. If different data sets lead to different
> >> values, then one could test different values in sensitivity analyses.
> >>
> >>
> >>
> >>
> >> Best,
> >> Lukasz
> >> --
> >> Lukasz Stasielowicz
> >> Osnabrück University
> >> Institute for Psychology
> >> Research methods, psychological assessment, and evaluation
> >> Seminarstraße 20
> >> 49074 Osnabrück (Germany)
> >>
> >> See also:
> >> On 03.08.2022 12:00, r-sig-meta-analysis-request using r-project.org wrote:
> >>> Send R-sig-meta-analysis mailing list submissions to
> >>> r-sig-meta-analysis using r-project.org
> >>>
> >>> To subscribe or unsubscribe via the World Wide Web, visit
> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >>> or, via email, send a message with subject or body 'help' to
> >>> r-sig-meta-analysis-request using r-project.org
> >>>
> >>> You can reach the person managing the list at
> >>> r-sig-meta-analysis-owner using r-project.org
> >>>
> >>> When replying, please edit your Subject line so it is more specific
> >>> than "Re: Contents of R-sig-meta-analysis digest..."
> >>>
> >>>
> >>> Today's Topics:
> >>>
> >>> 1. specification of phi in vcalc (Danielle Hiam)
> >>>
> >>> --------------------------------------------------------------------
> >>> --
> >>>
> >>> Message: 1
> >>> Date: Wed, 3 Aug 2022 06:54:48 +0000
> >>> From: Danielle Hiam <danielle.hiam using deakin.edu.au>
> >>> To: "r-sig-meta-analysis using r-project.org"
> >>> <r-sig-meta-analysis using r-project.org>
> >>> Subject: [R-meta] specification of phi in vcalc
> >>> Message-ID:
> >>> <
> >> ME3PR01MB54642B304670F1BF825E6F60B79C9 using ME3PR01MB5464.ausprd01.prod.out
> >> look.com
> >>>
> >>>
> >>> Content-Type: text/plain; charset="iso-8859-1"
> >>>
> >>> Hello,
> >>>
> >>> I am running a meta-analysis that has nested data and dependent
> >>> effect
> >> sizes (multiple timepoints). Therefore, I need to take into account
> >> effect sizes that are correlated and also cluster within studies. I
> >> will run vcalc, then the rma.mv function and then use robust with
> >> clubsandwich set to true to "correct" for any mis-specification of the
> >> model. However, I am looking for some clarification regarding how to
> >> "guess" the value for phi in the vcalc function, R code: V <-
> >> vcalc(FC_SD, cluster, time1=Time_NUM, data, phi=0.6).
> >>>
> >>> When I change phi from 0.6 to 0.8 in vcalc I get very different
> >>> results
> >> from the meta-analysis (rma.mv followed by robust(.... Clubsanwich=T).
> >> I have read James Pustejovsky paper (DOI:
> >> https://doi.org/10.1007/s11121-021-01246-3) on this, where he suggests
> >> that in situations where no information is available, the meta-analyst
> >> might pick a plausible value and then conduct sensitivity analysis
> >> across a range reasonable values. How would I go about this?
> >>>
> >>> Any information regarding specification of phi would be greatly
> >> appreciated
> >>>
> >>> Thanks,
> >>> Danielle
> >>>
> >>> Deakin University
> >>> Melbourne Burwood Campus, 221 Burwood Highway, Burwood VIC 3125
> >>> danielle.hiam using deakin.edu.au ipan.deakin.edu.au Deakin University
> >>> CRICOS Provider Code: 00113B
> >>>
> >>> Stay connected with IPAN
> >>> Facebook Twitter
> >>>
> >>> Important Notice: The contents of this email are intended solely for
> >>> the
> >> named addressee and are confidential; any unauthorised use,
> >> reproduction or storage of the contents is expressly prohibited.
> >>> If you have received this email in error, please delete it and any
> >> attachments immediately and advise the sender by return email or
> telephone.
> >> Deakin University does not warrant that this email and any attachments
> >> are error or virus free.
> >>>
> >>>
> >>>
> >>> Important Notice: The contents of this email are intended solely for
> >>> the
> >> named addressee and are confidential; any unauthorised use,
> >> reproduction or storage of the contents is expressly prohibited. If
> >> you have received this email in error, please delete it and any
> >> attachments immediately and advise the sender by return email or
> telephone.
> >>>
> >>> Deakin University does not warrant that this email and any
> >>> attachments
> >> are error or virus free.
> >>>
> >>>
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> Subject: Digest Footer
> >>>
> >>> _______________________________________________
> >>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> >>> To manage your subscription to this mailing list, go to:
> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >>>
> >>>
> >>> ------------------------------
> >>>
> >>> End of R-sig-meta-analysis Digest, Vol 63, Issue 1
> >>> **************************************************
> >>
> >> _______________________________________________
> >> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> >> To manage your subscription to this mailing list, go to:
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >>
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org To
> manage your subscription to this mailing list, go to:
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
> > Important Notice: The contents of this email are intended solely for the
> named addressee and are confidential; any unauthorised use, reproduction or
> storage of the contents is expressly prohibited. If you have received this
> email in error, please delete it and any attachments immediately and advise
> the sender by return email or telephone.
> >
> > Deakin University does not warrant that this email and any attachments
> are error or virus free.
> > _______________________________________________
> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> > To manage your subscription to this mailing list, go to:
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
> >
>
> --
> Michael
> http://www.dewey.myzen.co.uk/home.html
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list