[R-meta] SMD Metric

[Yuhang Hu]

Oops, correcting a typo in the second instance of x2:

x2 <- 40 + x2 * 3

Thank you,
Yuhang

On Sun, Apr 2, 2023 at 6:02 PM Yuhang Hu <yh342 using nau.edu> wrote:

> Thank you, James.
>
> I think we usually treat items on a test/instrument to be equivalent (all
> other things equal), if they differ from one another by some additive and
> multiplicative constants. We can probably extend this inter-item
> equivalency concept to equate different tests (all other things equal).
>
> It seems some effect sizes (e.g., SMD and correlation coef.) take this
> into account and 'work best' if such equivalence holds and work less well
> as this equivalence weakens.
>
> But other effect sizes seem to be somewhat insensitive to such
> equivalency. ROM seems to be one of them. But do you think that it is
> desirable for ROM to not give the same estimates for when such an
> equivalence, in fact, exists:
>
> x1 <- rnorm(50, 36, 6)
> x2 <- rnorm(50, 33, 6)
>
> library(metafor)
> escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>        m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>
> x1 <- 40 + x1 * 3
> x2 <- 40 + x2 * 7
>
> escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>        m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>
> Many thanks,
> Yuhang
>
> PS. With a lack of linear equatability, ROM may probably be preferred over
> SMD. However, I wonder what to replace COR with, when there is a lack of
> linear equatability? I would think in that case there is no alternative to
> COR and we need to resort to adding methodological moderators.
>
> On Sun, Apr 2, 2023 at 3:18 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Hi Yuhang,
>>
>> On the relationship between linear equatability and SMDs, this article
>> has a good discussion (and also see reference therein):
>>
>> Hedges, L. V. (2008). What are effect sizes and why do we need them?. *Child
>> development perspectives*, *2*(3), 167-171.
>> https://doi.org/10.1111/j.1750-8606.2008.00060.x
>>
>> Just to clarify, my post is NOT implying that the linear
>> equatability assumption is a requirement for using standardized mean
>> differences. I said only that it is "an ideal case" if linear
>> equatability holds. I agree with Wolfgang that, in practice, linear
>> equatability is unlikely to hold in the strict sense. The possible reasons
>> that you listed are all right on.
>>
>> James
>>
>> On Sun, Apr 2, 2023 at 4:00 PM Yuhang Hu via R-sig-meta-analysis <
>> r-sig-meta-analysis using r-project.org> wrote:
>>
>>> Dear Wolfgang,
>>>
>>> Thank you so much for your response. I would imagine that linear
>>> equatability is likely required for the use of many other effect sizes
>>> (e.g., correlation coefficients), right?
>>>
>>> But is/are there possibly some reference(s) discussing the 'linear
>>> equatability requirement' for the use of SMD or any other effect sizes
>>> (or
>>> perhaps any additional considerations like the ones I mention below)?
>>>
>>> You noted that "But many scales/instruments/questionnaires do not exhibit
>>> such strict linear equatability".
>>>
>>> I wonder what are the underlying reasons for that? For instance, the lack
>>> of linear equatability is because the instrument across studies (A) could
>>> target slightly different constructs (so their latent constructs differs
>>> in
>>> location and scale by a bit), or (B) they differ in length or time
>>> allowed
>>> to respond to the items (and thus in reliability), or (C) the items
>>> across
>>> the instruments differ in degrees of item difficulty and discrimination,
>>> or
>>> perhaps (D) the items across the instruments differ in their scale of
>>> measurement (one binary, another Likert scale etc.) and thus respondents'
>>> responses to the items across the instruments are distributed differently
>>> (one binomially distributed, another ordered-categorically distributed
>>> etc.)
>>>
>>> Thank you again, for your help,
>>> Yuhang
>>>
>>> On Sun, Apr 2, 2023 at 10:56 AM Viechtbauer, Wolfgang (NP) <
>>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>
>>> > Dear Yuhang,
>>> >
>>> > Essentially, it means that the values on one instrument are assumed to
>>> be
>>> > a linear transformation of the values on another instrument. For
>>> example,
>>> > say we have measured two groups using scale/instrument/questionnaire A
>>> and
>>> > we find:
>>> >
>>> > x1 <- rnorm(50, 36, 6)
>>> > x2 <- rnorm(50, 33, 6)
>>> >
>>> > library(metafor)
>>> > escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>>> >                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>>> >
>>> > Now imagine that instead of A, we had used another
>>> > scale/instrument/questionnaire B and that the values on that
>>> instrument are
>>> > simply a linear transformation of the scores that would have been
>>> obtained
>>> > on A:
>>> >
>>> > x1 <- 40 + x1 * 3
>>> > x2 <- 40 + x2 * 3
>>> >
>>> > escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>>> >                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>>> >
>>> > As you can see, the SMD values are identical then.
>>> >
>>> > So if values on different instruments are linearly equatable, then it
>>> > doesn't matter if we use A or B, the 'effect size' would be identical.
>>> >
>>> > But many scales/instruments/questionnaires do not exhibit such strict
>>> > linear equatability. In that case, SMD values may be systematically
>>> > higher/lower depending on the instrument used and we end up with a
>>> > measurement artifact in our meta-analysis.
>>> >
>>> > I hope that this clarifies things.
>>> >
>>> > Best,
>>> > Wolfgang
>>> >
>>> > >-----Original Message-----
>>> > >From: R-sig-meta-analysis [mailto:
>>> > r-sig-meta-analysis-bounces using r-project.org] On
>>> > >Behalf Of Yuhang Hu via R-sig-meta-analysis
>>> > >Sent: Sunday, 02 April, 2023 19:21
>>> > >To: R meta
>>> > >Cc: Yuhang Hu
>>> > >Subject: [R-meta] SMD Metric
>>> > >
>>> > >Hi Everyone,
>>> > >
>>> > >I had a question about the SMD effect size. I read on James' website
>>> that:
>>> > >
>>> > >"The ideal case for using the SMD metric is when the outcomes in
>>> different
>>> > >studies are linearly equatable. However, if outcomes exhibit
>>> mean-variance
>>> > >relationships, linearly equatability seems rather implausible."
>>> > >
>>> > >I was wondering what is meant by linear equatability in the outcomes
>>> in
>>> > >different studies and why is that needed for the use of SMD?  How
>>> could
>>> > the
>>> > >outcomes in different studies be perhaps non-linearly equatable or not
>>> > >equatable at all (neither linearly nor non-linearly)?
>>> > >
>>> > >(I also appreciate reference(s) that discuss such a requirement for
>>> the
>>> > use
>>> > >of the SMD metric)
>>> > >
>>> > >Thank you very much for your assistance,
>>> > >Yuhang
>>> >
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>> To manage your subscription to this mailing list, go to:
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>
>

	[[alternative HTML version deleted]]