[R-meta] guidance for modeling SMCC type effect size

[Reza Norouzian]

Dear Stefanou,

This has also come up many times on the list archived at:
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/ and on metafor's
website at: http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#three-level_model.
I encourage you to explore these two resources to learn more about
these modeling details.

In short, in your case, all the sources of heterogeneity before "id"
related to heterogeneity amoge true effect size aggregates. For
example, true effect sizes aggregated at the study level bouncing
around the overall mean, and true effect sizes aggregated at the
condition level bouncing around their respective study means).
Instead, ~ 1 | id accounts for heterogeneity between individual true
effects (not aggregates of any kind) that are nested in the immediate
level that contains them (in your case that immediate level is
condition).

For example, if in study 1, you have two rows coded T1 for condition,
one representing an effect estimate obtained at interval_id == 0, the
other obtained at interval_id == 1, you can potentially assume that
these two observed estimates are estimating different population
values perhaps by the virtue of being measured at different intervals
(i.e., the only remaining feature that distinguishes between them).
This is as narrow as it gets to assign random effects to a
meta-analytic dataset (i.e., to each row)

If there are enough studies like study 1 in your data (e.g., multiple
T1 rows with different id values due to interval_id), then, the
variance component due to id represents the heterogeneity in a given
condition, just like the variance component due to condition
represents the heterogeneity in a given study.

Best,
Reza






Reza



On Tue, Sep 14, 2021 at 6:33 PM Stefanou Revesz
<stefanourevesz using gmail.com> wrote:
>
> Very interesting! Thanks a lot! Just a very final question, what does
> the ~ 1 | id part do or assume?
>
> On Sun, Sep 12, 2021 at 6:13 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> >
> > I wonder how conditions could have their own autoregressive structure?
> >
> > At the study level, you could add *the same random-effect value* to
> > all the effects (i.e., rows) in each study. But within each study, at
> > the condition level, you could add *the same random-effect value* to
> > each set of effects (i.e., rows) that represent the same condition.
> >
> > Therefore, the two (study-level and condition-level) auto-regressive
> > structures may lead to different estimates of correlation; one
> > representing the common correlation among the adjacent interval_id
> > levels across studies, the other representing the common correlation
> > among the adjacent interval_id levels across the conditions nested in
> > studies.
> >
> > Likewise, the estimates of heterogeneity for SMCC effects at each
> > level of interval_id across the studies may be different from that
> > across the conditions nested in studies (assuming an HAR structure).
> >
> > why that could be necessary?
> >
> > There is nothing necessary, it's just an(other) assumption about the
> > structure of true effects, you can fit a model that employs such an
> > assumption and see if that improves the fit of the model to the data
> > relative to a model that doesn't utilize that assumption.
> >
> > Best,
> > Reza
> >
> >
> >
> >
> > On Sun, Sep 12, 2021 at 4:43 PM Stefanou Revesz
> > <stefanourevesz using gmail.com> wrote:
> > >
> > > Hi Reza,
> > >
> > > Thank you so much! A quick follow-up, when you say "If in each study,
> > > true SMCCs at all intervals across the conditions are assumed to have
> > > their own auto-regressively correlated structure as well", I wonder
> > > how conditions could have their own autoregressive structure and why
> > > that could be necessary?
> > >
> > > Thanks again!
> > >
> > > On Sun, Sep 12, 2021 at 2:57 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> > > >
> > > > Dear Stefanou,
> > > >
> > > > The modeling functions like rma.mv(), rma.uni() etc. don't generally
> > > > depend on the type of effect size. That aside, just to make sure, did
> > > > you compute the change from each pre-test to each follow-up post-test,
> > > > or the change from each testing occasion to the following one (e.g.,
> > > > pre-test to post-test1, post-test1 to post-test2 ...)?
> > > >
> > > > Regardless of how you define the intervals, if your studies include a
> > > > control (C) and multiple treatment conditions (T1, T2,...) that occur
> > > > across the studies, you can create a 'condition' variable to
> > > > distinguish between the control and the treatments' SMCCs (yi) in each
> > > > study over the intervals you have defined:
> > > >
> > > > study condition  yi  interval_id id
> > > > 1     T1        .1   0           1
> > > > 1     T1        .3   1           2
> > > > 1     T2        .7   0           3
> > > > 1     T2        .2   1           4
> > > > 1     C         .4   0           5
> > > > 1     C         .5   1           6
> > > > 2     T2        .6   0           7
> > > > 2     C         .9   1           8
> > > >
> > > > In which case, a starting point might be:
> > > >
> > > > rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
> > > > interval_id | study, ~ 1 | id), struct = "HAR")
> > > >
> > > > This model assumes that in each study, true SMCCs at all intervals are
> > > > auto-regressively correlated with each other regardless of the
> > > > conditions they belong to. If in each study, true SMCCs at all
> > > > intervals across the conditions are assumed to have their own
> > > > auto-regressively correlated structure as well, then, you can
> > > > consider:
> > > >
> > > > rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
> > > > interval_id | study, ~ interval_id | interaction(study,condition), ~ 1
> > > > | id), struct = c("HAR","HAR") )
> > > >
> > > > In both models, when the interaction term is decomposed to its simple
> > > > effects for each condition, you get the average SMCC for each
> > > > condition (e.g., T1, T2, or C) across the intervals for your studies.
> > > >
> > > > For your second type of effect size (y_i = SMCC_T - SMCC_C, v_i =
> > > > v_{i_{SMCC_T}} + v_{i_{SMCC_C}}), pretty much everything I said above
> > > > applies. However, in this case, you're basically modeling some sort of
> > > > simple effect for each treatment's change vs the control's change
> > > > across the intervals for your studies. So, by fitting the above model
> > > > but using this type of effect size, you'll be asking how such simple
> > > > effects change over the interval you have considered.
> > > >
> > > > I believe these metrics for effect size are nowadays less commonly
> > > > used, partly because you can use an SMD metric, which among other
> > > > things doesn't require direct knowledge of pre-post correlations for
> > > > their computation, model them using multivariate-multilevel models,
> > > > and present the results in perhaps more intuitive ways to your
> > > > audience.
> > > >
> > > > Does that help?
> > > >
> > > > Reza
> > > >
> > > >
> > > >
> > > > Reza
> > > >
> > > >
> > > > On Sun, Sep 12, 2021 at 6:15 AM Stefanou Revesz
> > > > <stefanourevesz using gmail.com> wrote:
> > > > >
> > > > >
> > > > > Dear Experts,
> > > > >
> > > > > I need some guidance for modeling two types of effect sizes (for two
> > > > > different meta-analyses) using the rma.mv() program in metafor.
> > > > >
> > > > > First, I have computed the standardized mean change (SMCC) between
> > > > > each pre-test and the follow-up post-tests in each of
> > > > > multiple-treatment studies.
> > > > >
> > > > > Second, I have computed the difference in standardized mean changes
> > > > > (SMCC) between a treatment and a control group at each pre-test to
> > > > > post-test intervals in each of multiple-treatment studies.
> > > > >
> > > > > The reason I want to use rma.mv() is that each of my studies could
> > > > > produce multiple of these types of effect sizes. But I wonder what
> > > > > would be a starting point for modeling these two types of effect size?
> > > > >
> > > > > Any help is appreciated,
> > > > > Stefanou
> > > > >
> > > > > _______________________________________________
> > > > > R-sig-meta-analysis mailing list
> > > > > R-sig-meta-analysis using r-project.org
> > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis