[R-meta] SMD Metric

[James Pustejovsky]

Hi Yuhang,

On the relationship between linear equatability and SMDs, this article has
a good discussion (and also see reference therein):

Hedges, L. V. (2008). What are effect sizes and why do we need them?. *Child
development perspectives*, *2*(3), 167-171.
https://doi.org/10.1111/j.1750-8606.2008.00060.x

Just to clarify, my post is NOT implying that the linear
equatability assumption is a requirement for using standardized mean
differences. I said only that it is "an ideal case" if linear
equatability holds. I agree with Wolfgang that, in practice, linear
equatability is unlikely to hold in the strict sense. The possible reasons
that you listed are all right on.

James

On Sun, Apr 2, 2023 at 4:00 PM Yuhang Hu via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Dear Wolfgang,
>
> Thank you so much for your response. I would imagine that linear
> equatability is likely required for the use of many other effect sizes
> (e.g., correlation coefficients), right?
>
> But is/are there possibly some reference(s) discussing the 'linear
> equatability requirement' for the use of SMD or any other effect sizes (or
> perhaps any additional considerations like the ones I mention below)?
>
> You noted that "But many scales/instruments/questionnaires do not exhibit
> such strict linear equatability".
>
> I wonder what are the underlying reasons for that? For instance, the lack
> of linear equatability is because the instrument across studies (A) could
> target slightly different constructs (so their latent constructs differs in
> location and scale by a bit), or (B) they differ in length or time allowed
> to respond to the items (and thus in reliability), or (C) the items across
> the instruments differ in degrees of item difficulty and discrimination, or
> perhaps (D) the items across the instruments differ in their scale of
> measurement (one binary, another Likert scale etc.) and thus respondents'
> responses to the items across the instruments are distributed differently
> (one binomially distributed, another ordered-categorically distributed
> etc.)
>
> Thank you again, for your help,
> Yuhang
>
> On Sun, Apr 2, 2023 at 10:56 AM Viechtbauer, Wolfgang (NP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> > Dear Yuhang,
> >
> > Essentially, it means that the values on one instrument are assumed to be
> > a linear transformation of the values on another instrument. For example,
> > say we have measured two groups using scale/instrument/questionnaire A
> and
> > we find:
> >
> > x1 <- rnorm(50, 36, 6)
> > x2 <- rnorm(50, 33, 6)
> >
> > library(metafor)
> > escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
> >                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
> >
> > Now imagine that instead of A, we had used another
> > scale/instrument/questionnaire B and that the values on that instrument
> are
> > simply a linear transformation of the scores that would have been
> obtained
> > on A:
> >
> > x1 <- 40 + x1 * 3
> > x2 <- 40 + x2 * 3
> >
> > escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
> >                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
> >
> > As you can see, the SMD values are identical then.
> >
> > So if values on different instruments are linearly equatable, then it
> > doesn't matter if we use A or B, the 'effect size' would be identical.
> >
> > But many scales/instruments/questionnaires do not exhibit such strict
> > linear equatability. In that case, SMD values may be systematically
> > higher/lower depending on the instrument used and we end up with a
> > measurement artifact in our meta-analysis.
> >
> > I hope that this clarifies things.
> >
> > Best,
> > Wolfgang
> >
> > >-----Original Message-----
> > >From: R-sig-meta-analysis [mailto:
> > r-sig-meta-analysis-bounces using r-project.org] On
> > >Behalf Of Yuhang Hu via R-sig-meta-analysis
> > >Sent: Sunday, 02 April, 2023 19:21
> > >To: R meta
> > >Cc: Yuhang Hu
> > >Subject: [R-meta] SMD Metric
> > >
> > >Hi Everyone,
> > >
> > >I had a question about the SMD effect size. I read on James' website
> that:
> > >
> > >"The ideal case for using the SMD metric is when the outcomes in
> different
> > >studies are linearly equatable. However, if outcomes exhibit
> mean-variance
> > >relationships, linearly equatability seems rather implausible."
> > >
> > >I was wondering what is meant by linear equatability in the outcomes in
> > >different studies and why is that needed for the use of SMD?  How could
> > the
> > >outcomes in different studies be perhaps non-linearly equatable or not
> > >equatable at all (neither linearly nor non-linearly)?
> > >
> > >(I also appreciate reference(s) that discuss such a requirement for the
> > use
> > >of the SMD metric)
> > >
> > >Thank you very much for your assistance,
> > >Yuhang
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]