[R-meta] guidance for modeling SMCC type effect size
Reza Norouzian
rnorouz|@n @end|ng |rom gm@||@com
Thu Sep 16 19:30:58 CEST 2021
No! That would mean that the true row effects in each study share the
same random-effect with the corresponding true row effects in all
other studies clearly for no systematic reason (e.g., a shared
condition) other than how you have arranged the rows for each study in
your dataset! So, this makes no sense as there are innumerable ways
you could have arranged the rows in each study.
On Thu, Sep 16, 2021 at 10:15 AM Stefanou Revesz
<stefanourevesz using gmail.com> wrote:
>
> Dear Reza,
>
> Thank you so much for the thorough explanation. Sorry for being a
> pain, but what happens if the id variable recycles through each study
> like for example in study 1, we have 1,2,3..., then again, in study 2
> we have 1,2,3,... and so on. Does ~ 1 | id still mean the same thing?
>
> On Wed, Sep 15, 2021 at 1:22 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> >
> > Dear Stefanou,
> >
> > This has also come up many times on the list archived at:
> > https://stat.ethz.ch/pipermail/r-sig-meta-analysis/ and on metafor's
> > website at: http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#three-level_model.
> > I encourage you to explore these two resources to learn more about
> > these modeling details.
> >
> > In short, in your case, all the sources of heterogeneity before "id"
> > related to heterogeneity amoge true effect size aggregates. For
> > example, true effect sizes aggregated at the study level bouncing
> > around the overall mean, and true effect sizes aggregated at the
> > condition level bouncing around their respective study means).
> > Instead, ~ 1 | id accounts for heterogeneity between individual true
> > effects (not aggregates of any kind) that are nested in the immediate
> > level that contains them (in your case that immediate level is
> > condition).
> >
> > For example, if in study 1, you have two rows coded T1 for condition,
> > one representing an effect estimate obtained at interval_id == 0, the
> > other obtained at interval_id == 1, you can potentially assume that
> > these two observed estimates are estimating different population
> > values perhaps by the virtue of being measured at different intervals
> > (i.e., the only remaining feature that distinguishes between them).
> > This is as narrow as it gets to assign random effects to a
> > meta-analytic dataset (i.e., to each row)
> >
> > If there are enough studies like study 1 in your data (e.g., multiple
> > T1 rows with different id values due to interval_id), then, the
> > variance component due to id represents the heterogeneity in a given
> > condition, just like the variance component due to condition
> > represents the heterogeneity in a given study.
> >
> > Best,
> > Reza
> >
> >
> >
> >
> >
> >
> > Reza
> >
> >
> >
> > On Tue, Sep 14, 2021 at 6:33 PM Stefanou Revesz
> > <stefanourevesz using gmail.com> wrote:
> > >
> > > Very interesting! Thanks a lot! Just a very final question, what does
> > > the ~ 1 | id part do or assume?
> > >
> > > On Sun, Sep 12, 2021 at 6:13 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> > > >
> > > > I wonder how conditions could have their own autoregressive structure?
> > > >
> > > > At the study level, you could add *the same random-effect value* to
> > > > all the effects (i.e., rows) in each study. But within each study, at
> > > > the condition level, you could add *the same random-effect value* to
> > > > each set of effects (i.e., rows) that represent the same condition.
> > > >
> > > > Therefore, the two (study-level and condition-level) auto-regressive
> > > > structures may lead to different estimates of correlation; one
> > > > representing the common correlation among the adjacent interval_id
> > > > levels across studies, the other representing the common correlation
> > > > among the adjacent interval_id levels across the conditions nested in
> > > > studies.
> > > >
> > > > Likewise, the estimates of heterogeneity for SMCC effects at each
> > > > level of interval_id across the studies may be different from that
> > > > across the conditions nested in studies (assuming an HAR structure).
> > > >
> > > > why that could be necessary?
> > > >
> > > > There is nothing necessary, it's just an(other) assumption about the
> > > > structure of true effects, you can fit a model that employs such an
> > > > assumption and see if that improves the fit of the model to the data
> > > > relative to a model that doesn't utilize that assumption.
> > > >
> > > > Best,
> > > > Reza
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Sep 12, 2021 at 4:43 PM Stefanou Revesz
> > > > <stefanourevesz using gmail.com> wrote:
> > > > >
> > > > > Hi Reza,
> > > > >
> > > > > Thank you so much! A quick follow-up, when you say "If in each study,
> > > > > true SMCCs at all intervals across the conditions are assumed to have
> > > > > their own auto-regressively correlated structure as well", I wonder
> > > > > how conditions could have their own autoregressive structure and why
> > > > > that could be necessary?
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > On Sun, Sep 12, 2021 at 2:57 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> > > > > >
> > > > > > Dear Stefanou,
> > > > > >
> > > > > > The modeling functions like rma.mv(), rma.uni() etc. don't generally
> > > > > > depend on the type of effect size. That aside, just to make sure, did
> > > > > > you compute the change from each pre-test to each follow-up post-test,
> > > > > > or the change from each testing occasion to the following one (e.g.,
> > > > > > pre-test to post-test1, post-test1 to post-test2 ...)?
> > > > > >
> > > > > > Regardless of how you define the intervals, if your studies include a
> > > > > > control (C) and multiple treatment conditions (T1, T2,...) that occur
> > > > > > across the studies, you can create a 'condition' variable to
> > > > > > distinguish between the control and the treatments' SMCCs (yi) in each
> > > > > > study over the intervals you have defined:
> > > > > >
> > > > > > study condition yi interval_id id
> > > > > > 1 T1 .1 0 1
> > > > > > 1 T1 .3 1 2
> > > > > > 1 T2 .7 0 3
> > > > > > 1 T2 .2 1 4
> > > > > > 1 C .4 0 5
> > > > > > 1 C .5 1 6
> > > > > > 2 T2 .6 0 7
> > > > > > 2 C .9 1 8
> > > > > >
> > > > > > In which case, a starting point might be:
> > > > > >
> > > > > > rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
> > > > > > interval_id | study, ~ 1 | id), struct = "HAR")
> > > > > >
> > > > > > This model assumes that in each study, true SMCCs at all intervals are
> > > > > > auto-regressively correlated with each other regardless of the
> > > > > > conditions they belong to. If in each study, true SMCCs at all
> > > > > > intervals across the conditions are assumed to have their own
> > > > > > auto-regressively correlated structure as well, then, you can
> > > > > > consider:
> > > > > >
> > > > > > rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
> > > > > > interval_id | study, ~ interval_id | interaction(study,condition), ~ 1
> > > > > > | id), struct = c("HAR","HAR") )
> > > > > >
> > > > > > In both models, when the interaction term is decomposed to its simple
> > > > > > effects for each condition, you get the average SMCC for each
> > > > > > condition (e.g., T1, T2, or C) across the intervals for your studies.
> > > > > >
> > > > > > For your second type of effect size (y_i = SMCC_T - SMCC_C, v_i =
> > > > > > v_{i_{SMCC_T}} + v_{i_{SMCC_C}}), pretty much everything I said above
> > > > > > applies. However, in this case, you're basically modeling some sort of
> > > > > > simple effect for each treatment's change vs the control's change
> > > > > > across the intervals for your studies. So, by fitting the above model
> > > > > > but using this type of effect size, you'll be asking how such simple
> > > > > > effects change over the interval you have considered.
> > > > > >
> > > > > > I believe these metrics for effect size are nowadays less commonly
> > > > > > used, partly because you can use an SMD metric, which among other
> > > > > > things doesn't require direct knowledge of pre-post correlations for
> > > > > > their computation, model them using multivariate-multilevel models,
> > > > > > and present the results in perhaps more intuitive ways to your
> > > > > > audience.
> > > > > >
> > > > > > Does that help?
> > > > > >
> > > > > > Reza
> > > > > >
> > > > > >
> > > > > >
> > > > > > Reza
> > > > > >
> > > > > >
> > > > > > On Sun, Sep 12, 2021 at 6:15 AM Stefanou Revesz
> > > > > > <stefanourevesz using gmail.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > > Dear Experts,
> > > > > > >
> > > > > > > I need some guidance for modeling two types of effect sizes (for two
> > > > > > > different meta-analyses) using the rma.mv() program in metafor.
> > > > > > >
> > > > > > > First, I have computed the standardized mean change (SMCC) between
> > > > > > > each pre-test and the follow-up post-tests in each of
> > > > > > > multiple-treatment studies.
> > > > > > >
> > > > > > > Second, I have computed the difference in standardized mean changes
> > > > > > > (SMCC) between a treatment and a control group at each pre-test to
> > > > > > > post-test intervals in each of multiple-treatment studies.
> > > > > > >
> > > > > > > The reason I want to use rma.mv() is that each of my studies could
> > > > > > > produce multiple of these types of effect sizes. But I wonder what
> > > > > > > would be a starting point for modeling these two types of effect size?
> > > > > > >
> > > > > > > Any help is appreciated,
> > > > > > > Stefanou
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > R-sig-meta-analysis mailing list
> > > > > > > R-sig-meta-analysis using r-project.org
> > > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
More information about the R-sig-meta-analysis
mailing list