[R-meta] guidance for modeling SMCC type effect size
Stefanou Revesz
@te|@noureve@z @end|ng |rom gm@||@com
Thu Sep 16 17:15:38 CEST 2021
Dear Reza,
Thank you so much for the thorough explanation. Sorry for being a
pain, but what happens if the id variable recycles through each study
like for example in study 1, we have 1,2,3..., then again, in study 2
we have 1,2,3,... and so on. Does ~ 1 | id still mean the same thing?
On Wed, Sep 15, 2021 at 1:22 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
>
> Dear Stefanou,
>
> This has also come up many times on the list archived at:
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/ and on metafor's
> website at: http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#three-level_model.
> I encourage you to explore these two resources to learn more about
> these modeling details.
>
> In short, in your case, all the sources of heterogeneity before "id"
> related to heterogeneity amoge true effect size aggregates. For
> example, true effect sizes aggregated at the study level bouncing
> around the overall mean, and true effect sizes aggregated at the
> condition level bouncing around their respective study means).
> Instead, ~ 1 | id accounts for heterogeneity between individual true
> effects (not aggregates of any kind) that are nested in the immediate
> level that contains them (in your case that immediate level is
> condition).
>
> For example, if in study 1, you have two rows coded T1 for condition,
> one representing an effect estimate obtained at interval_id == 0, the
> other obtained at interval_id == 1, you can potentially assume that
> these two observed estimates are estimating different population
> values perhaps by the virtue of being measured at different intervals
> (i.e., the only remaining feature that distinguishes between them).
> This is as narrow as it gets to assign random effects to a
> meta-analytic dataset (i.e., to each row)
>
> If there are enough studies like study 1 in your data (e.g., multiple
> T1 rows with different id values due to interval_id), then, the
> variance component due to id represents the heterogeneity in a given
> condition, just like the variance component due to condition
> represents the heterogeneity in a given study.
>
> Best,
> Reza
>
>
>
>
>
>
> Reza
>
>
>
> On Tue, Sep 14, 2021 at 6:33 PM Stefanou Revesz
> <stefanourevesz using gmail.com> wrote:
> >
> > Very interesting! Thanks a lot! Just a very final question, what does
> > the ~ 1 | id part do or assume?
> >
> > On Sun, Sep 12, 2021 at 6:13 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> > >
> > > I wonder how conditions could have their own autoregressive structure?
> > >
> > > At the study level, you could add *the same random-effect value* to
> > > all the effects (i.e., rows) in each study. But within each study, at
> > > the condition level, you could add *the same random-effect value* to
> > > each set of effects (i.e., rows) that represent the same condition.
> > >
> > > Therefore, the two (study-level and condition-level) auto-regressive
> > > structures may lead to different estimates of correlation; one
> > > representing the common correlation among the adjacent interval_id
> > > levels across studies, the other representing the common correlation
> > > among the adjacent interval_id levels across the conditions nested in
> > > studies.
> > >
> > > Likewise, the estimates of heterogeneity for SMCC effects at each
> > > level of interval_id across the studies may be different from that
> > > across the conditions nested in studies (assuming an HAR structure).
> > >
> > > why that could be necessary?
> > >
> > > There is nothing necessary, it's just an(other) assumption about the
> > > structure of true effects, you can fit a model that employs such an
> > > assumption and see if that improves the fit of the model to the data
> > > relative to a model that doesn't utilize that assumption.
> > >
> > > Best,
> > > Reza
> > >
> > >
> > >
> > >
> > > On Sun, Sep 12, 2021 at 4:43 PM Stefanou Revesz
> > > <stefanourevesz using gmail.com> wrote:
> > > >
> > > > Hi Reza,
> > > >
> > > > Thank you so much! A quick follow-up, when you say "If in each study,
> > > > true SMCCs at all intervals across the conditions are assumed to have
> > > > their own auto-regressively correlated structure as well", I wonder
> > > > how conditions could have their own autoregressive structure and why
> > > > that could be necessary?
> > > >
> > > > Thanks again!
> > > >
> > > > On Sun, Sep 12, 2021 at 2:57 PM Reza Norouzian <rnorouzian using gmail.com> wrote:
> > > > >
> > > > > Dear Stefanou,
> > > > >
> > > > > The modeling functions like rma.mv(), rma.uni() etc. don't generally
> > > > > depend on the type of effect size. That aside, just to make sure, did
> > > > > you compute the change from each pre-test to each follow-up post-test,
> > > > > or the change from each testing occasion to the following one (e.g.,
> > > > > pre-test to post-test1, post-test1 to post-test2 ...)?
> > > > >
> > > > > Regardless of how you define the intervals, if your studies include a
> > > > > control (C) and multiple treatment conditions (T1, T2,...) that occur
> > > > > across the studies, you can create a 'condition' variable to
> > > > > distinguish between the control and the treatments' SMCCs (yi) in each
> > > > > study over the intervals you have defined:
> > > > >
> > > > > study condition yi interval_id id
> > > > > 1 T1 .1 0 1
> > > > > 1 T1 .3 1 2
> > > > > 1 T2 .7 0 3
> > > > > 1 T2 .2 1 4
> > > > > 1 C .4 0 5
> > > > > 1 C .5 1 6
> > > > > 2 T2 .6 0 7
> > > > > 2 C .9 1 8
> > > > >
> > > > > In which case, a starting point might be:
> > > > >
> > > > > rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
> > > > > interval_id | study, ~ 1 | id), struct = "HAR")
> > > > >
> > > > > This model assumes that in each study, true SMCCs at all intervals are
> > > > > auto-regressively correlated with each other regardless of the
> > > > > conditions they belong to. If in each study, true SMCCs at all
> > > > > intervals across the conditions are assumed to have their own
> > > > > auto-regressively correlated structure as well, then, you can
> > > > > consider:
> > > > >
> > > > > rma.mv(yi ~ condition*interval_id, V = Some_V_matrix, random = list(~
> > > > > interval_id | study, ~ interval_id | interaction(study,condition), ~ 1
> > > > > | id), struct = c("HAR","HAR") )
> > > > >
> > > > > In both models, when the interaction term is decomposed to its simple
> > > > > effects for each condition, you get the average SMCC for each
> > > > > condition (e.g., T1, T2, or C) across the intervals for your studies.
> > > > >
> > > > > For your second type of effect size (y_i = SMCC_T - SMCC_C, v_i =
> > > > > v_{i_{SMCC_T}} + v_{i_{SMCC_C}}), pretty much everything I said above
> > > > > applies. However, in this case, you're basically modeling some sort of
> > > > > simple effect for each treatment's change vs the control's change
> > > > > across the intervals for your studies. So, by fitting the above model
> > > > > but using this type of effect size, you'll be asking how such simple
> > > > > effects change over the interval you have considered.
> > > > >
> > > > > I believe these metrics for effect size are nowadays less commonly
> > > > > used, partly because you can use an SMD metric, which among other
> > > > > things doesn't require direct knowledge of pre-post correlations for
> > > > > their computation, model them using multivariate-multilevel models,
> > > > > and present the results in perhaps more intuitive ways to your
> > > > > audience.
> > > > >
> > > > > Does that help?
> > > > >
> > > > > Reza
> > > > >
> > > > >
> > > > >
> > > > > Reza
> > > > >
> > > > >
> > > > > On Sun, Sep 12, 2021 at 6:15 AM Stefanou Revesz
> > > > > <stefanourevesz using gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > > Dear Experts,
> > > > > >
> > > > > > I need some guidance for modeling two types of effect sizes (for two
> > > > > > different meta-analyses) using the rma.mv() program in metafor.
> > > > > >
> > > > > > First, I have computed the standardized mean change (SMCC) between
> > > > > > each pre-test and the follow-up post-tests in each of
> > > > > > multiple-treatment studies.
> > > > > >
> > > > > > Second, I have computed the difference in standardized mean changes
> > > > > > (SMCC) between a treatment and a control group at each pre-test to
> > > > > > post-test intervals in each of multiple-treatment studies.
> > > > > >
> > > > > > The reason I want to use rma.mv() is that each of my studies could
> > > > > > produce multiple of these types of effect sizes. But I wonder what
> > > > > > would be a starting point for modeling these two types of effect size?
> > > > > >
> > > > > > Any help is appreciated,
> > > > > > Stefanou
> > > > > >
> > > > > > _______________________________________________
> > > > > > R-sig-meta-analysis mailing list
> > > > > > R-sig-meta-analysis using r-project.org
> > > > > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
More information about the R-sig-meta-analysis
mailing list