[R-meta] Equivalence testing for subgroup analysis in MA - TOSTER package
Lukasz Stasielowicz
|uk@@z@@t@@|e|ow|cz @end|ng |rom un|-o@n@brueck@de
Sat Oct 16 18:26:56 CEST 2021
Dear Jorge,
since you haven't received a response yet I will mention an alternative
approach. Unfortunately, I currently don't have time to check the TOSTER
package (new teaching period is starting soon).
As Daniel Lakens (2017) points out one can simply use 90% confidence
intervals for equivalence testing. If you are conducting a two-level
meta-analysis using MD or SMD then one could use the rma() function of
the metafor package and specify the desired confidence interval using
the level argument: rma(...,level=90). Next you can compare the
confidence interval printed in the output with your equivalence bounds.
If the confidence interval is within the equivalence bounds (and doesn't
touch them) then it can be interpreted as evidence for the equivalence
hypothesis.
Lakens, D. (2017). Equivalence tests: A practical primer for t tests,
correlations, and meta-analyses. Social Psychological and Personality
Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177
best,
Lukasz
--
Lukasz Stasielowicz
Osnabrück University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Seminarstraße 20
49074 Osnabrück (Germany)
Am 10.10.2021 um 19:13 schrieb r-sig-meta-analysis-request using r-project.org:
> Send R-sig-meta-analysis mailing list submissions to
> r-sig-meta-analysis using r-project.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> or, via email, send a message with subject or body 'help' to
> r-sig-meta-analysis-request using r-project.org
>
> You can reach the person managing the list at
> r-sig-meta-analysis-owner using r-project.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-meta-analysis digest..."
>
>
> Today's Topics:
>
> 1. Equivalence testing for subgroup analysis in MA - TOSTER
> package (Jorge Teixeira)
> 2. Re: Time as indicator vs time as meaning
> (Viechtbauer, Wolfgang (SP))
> 3. Re: Time as indicator vs time as meaning (Stefanou Revesz)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 10 Oct 2021 14:19:52 +0100
> From: Jorge Teixeira <jorgemmtteixeira using gmail.com>
> To: R meta <r-sig-meta-analysis using r-project.org>
> Subject: [R-meta] Equivalence testing for subgroup analysis in MA -
> TOSTER package
> Message-ID:
> <CAOYO_yA4x8K=xzfqcCKhEN5oPWa4V7oveWMci-LP9DdMhfCLmA using mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear All,
>
> Suppose someone is interested in doing equivalence testing in a MA for a
> subgroup analysis, for example, men vs women.
>
> The code used below is from the TOSTER package.
>
> *1)* If one is using SMD as the ES, one can use meta-regression and enter
> the coefficient and SE with the following formula, correct?
>
> TOSTER::TOSTmeta
> TOSTmeta(ES, var, se, low_eqbound_d, high_eqbound_d, alpha, plot = TRUE,
> verbose = TRUE)
>
>
> *2.1)* But what if the ES is MD? The previous code only works for SMD.
>
> Can we calculate the SD for each subgroup using a formula (*95%CI: dividing
> the length of the confidence interval by 3.92, and then multiplying by the
> square root of the sample size*) and then plugging in the following code
> for comparing 2 groups (raw values)?
>
> TOSTER::TOSTtwo.raw
> TOSTtwo.raw(m1, m2, sd1, sd2, n1, n2, low_eqbound, high_eqbound, alpha,
> var.equal, plot = TRUE, verbose = TRUE)
>
> *2.1*) Entering the tau2 instead SD directly in the previous formula would
> be incorrect?
>
> *3) *Any other alternative?
>
> Thanks and regards,
> JT
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 10 Oct 2021 13:56:30 +0000
> From: "Viechtbauer, Wolfgang (SP)"
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> To: Stefanou Revesz <stefanourevesz using gmail.com>
> Cc: R meta <r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Time as indicator vs time as meaning
> Message-ID: <4713f848dd2349a7a6862ae5d234948a using UM-MAIL3214.unimaas.nl>
> Content-Type: text/plain; charset="utf-8"
>
> See below for my responses.
>
> Best,
> Wolfgang
>
>> -----Original Message-----
>> From: Stefanou Revesz [mailto:stefanourevesz using gmail.com]
>> Sent: Saturday, 09 October, 2021 19:28
>> To: Viechtbauer, Wolfgang (SP)
>> Cc: Michael Dewey; R meta
>> Subject: Re: [R-meta] Time as indicator vs time as meaning
>>
>> Thank you so much, Wolfgang and Michael. So, based on your advice I
>> will use the following:
>>
>> rma.mv(es ~ time_meaning_wks, random = ~ time_meaning_wks | study,
>> struct = "CAR")
>>
>> Then I think, the fixed coef. of "time_meaning_wks" is interpreted as:
>> The change in true effects for 1 week change in time regardless of the
>> testing occasions' indicator.
>
> Correct. And now comes the part again where I am being pedantic:
>
> 1) To be precise: The average change.
>
> 2) You could run this model even if not a single study had used two (or more) measurement occasions (one could then not estimate rho - so in principle, the model is then overparameterized, but this isn't relevant to the point I am trying to make here). In this case, the coefficient for time_meaning_wks is estimated purely by examining the size of the effect across studies that used different measurement weeks. In this case, speaking of 'change' might suggest that the studies are actually providing evidence about the change in the effect over time, when in fact the evidence is purely cross-sectional (or "cross-studinal" if there even is such a word - Google gives me 0 hits for this, so maybe not, but then I just invented it). You *do* have studies that used multiple measurement occasions, so the coefficient for time_meaning_wks is actually a mixture of cross-studinal evidence and actual changes in the effect that occurred within studies. To disentangle such between- and within-study e
> vidence, one can apply methods that are well-known in the multilevel / longitudinal data analysis context, namely by computing the study-level means of time_meaning_wks and the deviations from these means within studies and then including both of these variables in the model.
>
>> One other option that we came up with to keep time_id (as a factor) in
>> the model was to control for "time_meaning_wks" as in:
>>
>> rma.mv(es ~ factor(time_id) + time_meaning_wks, random = ~
>> factor(time_id) | study, struct = "UN")
>>
>> Does this alternative make sense?
>
> One can do this, but it's trying to squeeze a lot of information out of your data. The factor(time_id) fixed effect is asking about differences in measurement occasions irrespective of their actual time values (so the second measurement occasions might be at 6 weeks in study 1 and at 24 weeks in study 2 but these are both treated identically). The time_meaning_wks effect is asking about (linear) changes (or differences -- see above) in the effect over time measured in weeks.
>
> And as discussed in a previous post, using something like 'random = ~ factor(time_id) | study, struct = "UN"' is likely to lead to a rather large number of variance components and correlations being estimated.
>
>> Thank you for your expertise,
>> Stefanou
>>
>> On Sat, Oct 9, 2021 at 11:13 AM Viechtbauer, Wolfgang (SP)
>> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>
>>> To add to this:
>>>
>>> 2. Terms used in 'random' are not allowed to have missing values in rma.mv(),
>> so those rows will need to be filtered out first before fitting the model.
>>>
>>> 3. rho in "CAR" is the autocorrelation for a one-unit difference in the time
>> variable. So if time is measured in weeks, then rho reflects the correlation
>> between two time points one week apart.
>>>
>>> Best,
>>> Wolfgang
>>>
>>>> -----Original Message-----
>>>> From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]
>>>> Sent: Saturday, 09 October, 2021 17:36
>>>> To: Stefanou Revesz; Viechtbauer, Wolfgang (SP)
>>>> Cc: R meta
>>>> Subject: Re: [R-meta] Time as indicator vs time as meaning
>>>>
>>>> Comments in-line
>>>>
>>>> On 09/10/2021 15:56, Stefanou Revesz wrote:
>>>>> Dear Wolfgang,
>>>>>
>>>>> Thank you for your reply. The rma.mv() documentation for CAR says:
>>>>> "the values of the "inner" variable should reflect the exact time
>>>>> points of the measurement".
>>>>>
>>>>> 1) Does that mean I should use: "time_meaning_wks | study" OR
>>>>> "time_id | study"?
>>>>
>>>> Use the continuous one time_meaning_wks
>>>>
>>>>> 2) Can I have missing in "time_meaning_wks"?
>>>>
>>>> I assume it will work, just try it, nothing will break.
>>>>
>>>>> 3) Do you possibly have a demonstration showing how to interpret CAR
>>>>> (or any other useful references to read about CAR)?
>>>>
>>>> If you type auto-regressive models into your favourite search engine you
>>>> should find plenty of material. There are a couple of examples of AR
>>>> models in the documentation, see ?rma.mv but neither of them is for a
>>>> continuous covariate.
>>>>
>>>>> Thank you very much,
>>>>> Stefanou
>>>>>
>>>>> On Sat, Oct 9, 2021 at 7:52 AM Viechtbauer, Wolfgang (SP)
>>>>> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>>>>
>>>>>> Indeed. But then struct="CAR" would probably be more
>> appropriate/parsimonious,
>>>> since "UN" will estimate a different tau^2 for every unique week value and a
>>>> different correlation for every possible pair of week values.
>>>>>>
>>>>>> Best,
>>>>>> Wolfgang
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>> project.org]
>>>> On
>>>>>>> Behalf Of Michael Dewey
>>>>>>> Sent: Saturday, 09 October, 2021 12:59
>>>>>>> To: Stefanou Revesz; R meta
>>>>>>> Subject: Re: [R-meta] Time as indicator vs time as meaning
>>>>>>>
>>>>>>> Dear Stefanou
>>>>>>>
>>>>>>> I think it would be find to use the continuous version both as fixed and
>>>>>>> random effect.
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>> On 09/10/2021 05:49, Stefanou Revesz wrote:
>>>>>>>> Dear Meta-Analysis Colleagues,
>>>>>>>>
>>>>>>>> We are meta-analyzing 73 longitudinal studies. But we have doubts
>>>>>>>> amongst us regarding how to combine the longitudinal effects of these
>>>>>>>> studies.
>>>>>>>>
>>>>>>>> On the one hand, if we use time only as an indicator of testing
>>>>>>>> occasions (pre-test and post-tests), and then use it as fixed and
>>>>>>>> random-effect as in:
>>>>>>>>
>>>>>>>> rma.mv(es ~ time_id, random = ~ time_id | study, struct = "UN")
>>>>>>>>
>>>>>>>> then, we have longitudinally combined apples and oranges. That is,
>>>>>>>> time 1 in one study may have covered six months, but time 1 in another
>>>>>>>> study may have covered 6 days. This, we think, is problematic in terms
>>>>>>>> of the interpretation of both the fixed and random-effects of time.
>>>>>>>>
>>>>>>>> So, we have coded for both time_id (testing occasions indicator) and
>>>>>>>> time_meaning_wks (length of actual time up to each testing occasion in
>>>>>>>> weeks).
>>>>>>>>
>>>>>>>> We are wondering how we should incorporate time_meaning_wks into our
>> model?
>>>>>>>>
>>>>>>>> Any help is appreciated,
>>>>>>>> Stefanou
>>>>>>>>
>>>>>>>> study time_id time_meaning_wks
>>>>>>>> 1 0 0
>>>>>>>> 1 1 4
>>>>>>>> 1 2 6
>>>>>>>> 2 0 0
>>>>>>>> 2 1 1
>
>
> ------------------------------
>
> Message: 3
> Date: Sun, 10 Oct 2021 12:13:03 -0500
> From: Stefanou Revesz <stefanourevesz using gmail.com>
> To: "Viechtbauer, Wolfgang (SP)"
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Cc: R meta <r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Time as indicator vs time as meaning
> Message-ID:
> <CA+Tgogr_muHr-zrhwSUuq0sPkRvH=-1uvoopon5LOK+Hc_zh7Q using mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear Wolfgang,
>
> This was such an important point. I'm highly grateful for this. To
> follow your advice, I'm going to do:
>
> rma.mv(es ~ time_wks_btw + time_wks_wth, random = ~ time_meaning_wks |
> study, struct = "CAR")
>
> But:
>
> 1) Can the right side of | in the random part, remain as
> "time_meaning_wks" or it should be "time_wks_wth"?
>
> 2) If I want to interact "condition" with time, which time moderator
> should I use: "time_wks_btw" or "time_wks_wth"?
>
> Many thanks,
> Stefanou
>
>
>
> On Sun, Oct 10, 2021 at 8:56 AM Viechtbauer, Wolfgang (SP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>
>> See below for my responses.
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: Stefanou Revesz [mailto:stefanourevesz using gmail.com]
>>> Sent: Saturday, 09 October, 2021 19:28
>>> To: Viechtbauer, Wolfgang (SP)
>>> Cc: Michael Dewey; R meta
>>> Subject: Re: [R-meta] Time as indicator vs time as meaning
>>>
>>> Thank you so much, Wolfgang and Michael. So, based on your advice I
>>> will use the following:
>>>
>>> rma.mv(es ~ time_meaning_wks, random = ~ time_meaning_wks | study,
>>> struct = "CAR")
>>>
>>> Then I think, the fixed coef. of "time_meaning_wks" is interpreted as:
>>> The change in true effects for 1 week change in time regardless of the
>>> testing occasions' indicator.
>>
>> Correct. And now comes the part again where I am being pedantic:
>>
>> 1) To be precise: The average change.
>>
>> 2) You could run this model even if not a single study had used two (or more) measurement occasions (one could then not estimate rho - so in principle, the model is then overparameterized, but this isn't relevant to the point I am trying to make here). In this case, the coefficient for time_meaning_wks is estimated purely by examining the size of the effect across studies that used different measurement weeks. In this case, speaking of 'change' might suggest that the studies are actually providing evidence about the change in the effect over time, when in fact the evidence is purely cross-sectional (or "cross-studinal" if there even is such a word - Google gives me 0 hits for this, so maybe not, but then I just invented it). You *do* have studies that used multiple measurement occasions, so the coefficient for time_meaning_wks is actually a mixture of cross-studinal evidence and actual changes in the effect that occurred within studies. To disentangle such between- and within-study
> evidence, one can apply methods that are well-known in the multilevel / longitudinal data analysis context, namely by computing the study-level means of time_meaning_wks and the deviations from these means within studies and then including both of these variables in the model.
>>
>>> One other option that we came up with to keep time_id (as a factor) in
>>> the model was to control for "time_meaning_wks" as in:
>>>
>>> rma.mv(es ~ factor(time_id) + time_meaning_wks, random = ~
>>> factor(time_id) | study, struct = "UN")
>>>
>>> Does this alternative make sense?
>>
>> One can do this, but it's trying to squeeze a lot of information out of your data. The factor(time_id) fixed effect is asking about differences in measurement occasions irrespective of their actual time values (so the second measurement occasions might be at 6 weeks in study 1 and at 24 weeks in study 2 but these are both treated identically). The time_meaning_wks effect is asking about (linear) changes (or differences -- see above) in the effect over time measured in weeks.
>>
>> And as discussed in a previous post, using something like 'random = ~ factor(time_id) | study, struct = "UN"' is likely to lead to a rather large number of variance components and correlations being estimated.
>>
>>> Thank you for your expertise,
>>> Stefanou
>>>
>>> On Sat, Oct 9, 2021 at 11:13 AM Viechtbauer, Wolfgang (SP)
>>> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>>
>>>> To add to this:
>>>>
>>>> 2. Terms used in 'random' are not allowed to have missing values in rma.mv(),
>>> so those rows will need to be filtered out first before fitting the model.
>>>>
>>>> 3. rho in "CAR" is the autocorrelation for a one-unit difference in the time
>>> variable. So if time is measured in weeks, then rho reflects the correlation
>>> between two time points one week apart.
>>>>
>>>> Best,
>>>> Wolfgang
>>>>
>>>>> -----Original Message-----
>>>>> From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]
>>>>> Sent: Saturday, 09 October, 2021 17:36
>>>>> To: Stefanou Revesz; Viechtbauer, Wolfgang (SP)
>>>>> Cc: R meta
>>>>> Subject: Re: [R-meta] Time as indicator vs time as meaning
>>>>>
>>>>> Comments in-line
>>>>>
>>>>> On 09/10/2021 15:56, Stefanou Revesz wrote:
>>>>>> Dear Wolfgang,
>>>>>>
>>>>>> Thank you for your reply. The rma.mv() documentation for CAR says:
>>>>>> "the values of the "inner" variable should reflect the exact time
>>>>>> points of the measurement".
>>>>>>
>>>>>> 1) Does that mean I should use: "time_meaning_wks | study" OR
>>>>>> "time_id | study"?
>>>>>
>>>>> Use the continuous one time_meaning_wks
>>>>>
>>>>>> 2) Can I have missing in "time_meaning_wks"?
>>>>>
>>>>> I assume it will work, just try it, nothing will break.
>>>>>
>>>>>> 3) Do you possibly have a demonstration showing how to interpret CAR
>>>>>> (or any other useful references to read about CAR)?
>>>>>
>>>>> If you type auto-regressive models into your favourite search engine you
>>>>> should find plenty of material. There are a couple of examples of AR
>>>>> models in the documentation, see ?rma.mv but neither of them is for a
>>>>> continuous covariate.
>>>>>
>>>>>> Thank you very much,
>>>>>> Stefanou
>>>>>>
>>>>>> On Sat, Oct 9, 2021 at 7:52 AM Viechtbauer, Wolfgang (SP)
>>>>>> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>>>>>
>>>>>>> Indeed. But then struct="CAR" would probably be more
>>> appropriate/parsimonious,
>>>>> since "UN" will estimate a different tau^2 for every unique week value and a
>>>>> different correlation for every possible pair of week values.
>>>>>>>
>>>>>>> Best,
>>>>>>> Wolfgang
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>>> project.org]
>>>>> On
>>>>>>>> Behalf Of Michael Dewey
>>>>>>>> Sent: Saturday, 09 October, 2021 12:59
>>>>>>>> To: Stefanou Revesz; R meta
>>>>>>>> Subject: Re: [R-meta] Time as indicator vs time as meaning
>>>>>>>>
>>>>>>>> Dear Stefanou
>>>>>>>>
>>>>>>>> I think it would be find to use the continuous version both as fixed and
>>>>>>>> random effect.
>>>>>>>>
>>>>>>>> Michael
>>>>>>>>
>>>>>>>> On 09/10/2021 05:49, Stefanou Revesz wrote:
>>>>>>>>> Dear Meta-Analysis Colleagues,
>>>>>>>>>
>>>>>>>>> We are meta-analyzing 73 longitudinal studies. But we have doubts
>>>>>>>>> amongst us regarding how to combine the longitudinal effects of these
>>>>>>>>> studies.
>>>>>>>>>
>>>>>>>>> On the one hand, if we use time only as an indicator of testing
>>>>>>>>> occasions (pre-test and post-tests), and then use it as fixed and
>>>>>>>>> random-effect as in:
>>>>>>>>>
>>>>>>>>> rma.mv(es ~ time_id, random = ~ time_id | study, struct = "UN")
>>>>>>>>>
>>>>>>>>> then, we have longitudinally combined apples and oranges. That is,
>>>>>>>>> time 1 in one study may have covered six months, but time 1 in another
>>>>>>>>> study may have covered 6 days. This, we think, is problematic in terms
>>>>>>>>> of the interpretation of both the fixed and random-effects of time.
>>>>>>>>>
>>>>>>>>> So, we have coded for both time_id (testing occasions indicator) and
>>>>>>>>> time_meaning_wks (length of actual time up to each testing occasion in
>>>>>>>>> weeks).
>>>>>>>>>
>>>>>>>>> We are wondering how we should incorporate time_meaning_wks into our
>>> model?
>>>>>>>>>
>>>>>>>>> Any help is appreciated,
>>>>>>>>> Stefanou
>>>>>>>>>
>>>>>>>>> study time_id time_meaning_wks
>>>>>>>>> 1 0 0
>>>>>>>>> 1 1 4
>>>>>>>>> 1 2 6
>>>>>>>>> 2 0 0
>>>>>>>>> 2 1 1
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>
> ------------------------------
>
> End of R-sig-meta-analysis Digest, Vol 53, Issue 37
> ***************************************************
>
More information about the R-sig-meta-analysis
mailing list