[R-meta] SMD Metric

Mon Apr 3 03:02:25 CEST 2023

Thank you, James.

I think we usually treat items on a test/instrument to be equivalent (all
other things equal), if they differ from one another by some additive and
multiplicative constants. We can probably extend this inter-item
equivalency concept to equate different tests (all other things equal).

It seems some effect sizes (e.g., SMD and correlation coef.) take this into
account and 'work best' if such equivalence holds and work less well as
this equivalence weakens.

But other effect sizes seem to be somewhat insensitive to such equivalency.
ROM seems to be one of them. But do you think that it is desirable for ROM
to not give the same estimates for when such an equivalence, in fact,
exists:

x1 <- rnorm(50, 36, 6)
x2 <- rnorm(50, 33, 6)

library(metafor)
escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))

x1 <- 40 + x1 * 3
x2 <- 40 + x2 * 7

escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))

Many thanks,
Yuhang

PS. With a lack of linear equatability, ROM may probably be preferred over
SMD. However, I wonder what to replace COR with, when there is a lack of
linear equatability? I would think in that case there is no alternative to
COR and we need to resort to adding methodological moderators.

On Sun, Apr 2, 2023 at 3:18 PM James Pustejovsky <jepusto using gmail.com> wrote:

> Hi Yuhang,
>
> On the relationship between linear equatability and SMDs, this article has
> a good discussion (and also see reference therein):
>
> Hedges, L. V. (2008). What are effect sizes and why do we need them?. *Child
> development perspectives*, *2*(3), 167-171.
> https://doi.org/10.1111/j.1750-8606.2008.00060.x
>
> Just to clarify, my post is NOT implying that the linear
> equatability assumption is a requirement for using standardized mean
> differences. I said only that it is "an ideal case" if linear
> equatability holds. I agree with Wolfgang that, in practice, linear
> equatability is unlikely to hold in the strict sense. The possible reasons
> that you listed are all right on.
>
> James
>
> On Sun, Apr 2, 2023 at 4:00 PM Yuhang Hu via R-sig-meta-analysis <
> r-sig-meta-analysis using r-project.org> wrote:
>
>> Dear Wolfgang,
>>
>> Thank you so much for your response. I would imagine that linear
>> equatability is likely required for the use of many other effect sizes
>> (e.g., correlation coefficients), right?
>>
>> But is/are there possibly some reference(s) discussing the 'linear
>> equatability requirement' for the use of SMD or any other effect sizes (or
>> perhaps any additional considerations like the ones I mention below)?
>>
>> You noted that "But many scales/instruments/questionnaires do not exhibit
>> such strict linear equatability".
>>
>> I wonder what are the underlying reasons for that? For instance, the lack
>> of linear equatability is because the instrument across studies (A) could
>> target slightly different constructs (so their latent constructs differs
>> in
>> location and scale by a bit), or (B) they differ in length or time allowed
>> to respond to the items (and thus in reliability), or (C) the items across
>> the instruments differ in degrees of item difficulty and discrimination,
>> or
>> perhaps (D) the items across the instruments differ in their scale of
>> measurement (one binary, another Likert scale etc.) and thus respondents'
>> responses to the items across the instruments are distributed differently
>> (one binomially distributed, another ordered-categorically distributed
>> etc.)
>>
>> Thank you again, for your help,
>> Yuhang
>>
>> On Sun, Apr 2, 2023 at 10:56 AM Viechtbauer, Wolfgang (NP) <
>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>
>> > Dear Yuhang,
>> >
>> > Essentially, it means that the values on one instrument are assumed to
>> be
>> > a linear transformation of the values on another instrument. For
>> example,
>> > say we have measured two groups using scale/instrument/questionnaire A
>> and
>> > we find:
>> >
>> > x1 <- rnorm(50, 36, 6)
>> > x2 <- rnorm(50, 33, 6)
>> >
>> > library(metafor)
>> > escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>> >                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>> >
>> > Now imagine that instead of A, we had used another
>> > scale/instrument/questionnaire B and that the values on that instrument
>> are
>> > simply a linear transformation of the scores that would have been
>> obtained
>> > on A:
>> >
>> > x1 <- 40 + x1 * 3
>> > x2 <- 40 + x2 * 3
>> >
>> > escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>> >                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>> >
>> > As you can see, the SMD values are identical then.
>> >
>> > So if values on different instruments are linearly equatable, then it
>> > doesn't matter if we use A or B, the 'effect size' would be identical.
>> >
>> > But many scales/instruments/questionnaires do not exhibit such strict
>> > linear equatability. In that case, SMD values may be systematically
>> > higher/lower depending on the instrument used and we end up with a
>> > measurement artifact in our meta-analysis.
>> >
>> > I hope that this clarifies things.
>> >
>> > Best,
>> > Wolfgang
>> >
>> > >-----Original Message-----
>> > >From: R-sig-meta-analysis [mailto:
>> > r-sig-meta-analysis-bounces using r-project.org] On
>> > >Behalf Of Yuhang Hu via R-sig-meta-analysis
>> > >Sent: Sunday, 02 April, 2023 19:21
>> > >To: R meta
>> > >Cc: Yuhang Hu
>> > >Subject: [R-meta] SMD Metric
>> > >
>> > >Hi Everyone,
>> > >
>> > >I had a question about the SMD effect size. I read on James' website
>> that:
>> > >
>> > >"The ideal case for using the SMD metric is when the outcomes in
>> different
>> > >studies are linearly equatable. However, if outcomes exhibit
>> mean-variance
>> > >relationships, linearly equatability seems rather implausible."
>> > >
>> > >I was wondering what is meant by linear equatability in the outcomes in
>> > >different studies and why is that needed for the use of SMD?  How could
>> > the
>> > >outcomes in different studies be perhaps non-linearly equatable or not
>> > >equatable at all (neither linearly nor non-linearly)?
>> > >
>> > >(I also appreciate reference(s) that discuss such a requirement for the
>> > use
>> > >of the SMD metric)
>> > >
>> > >Thank you very much for your assistance,
>> > >Yuhang
>> >
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

	[[alternative HTML version deleted]]