[R-meta] SMD Metric

[Yuhang Hu]

Sure, thanks. I think the trouble in my meta-analysis is that all studies
report means and sds, but these means and sds come from instruments that
likely differ in ways that I described in my previous post.

I'm sure for some instruments (perhaps the ones' using Likert scale) SMD
may be initially preferred over ROM, but for others ROM might be a better
choice as the means and sds often refer to counts or portions of counts.

The other thing is that even for those that seem to initially align better
with SMD, the cross-instrument differences that I described in my previous
post could potentially outweigh that SMD alignment such that some
methodologically induced heterogeneity may be introduced to the
meta-analysis of SMDs.

I may eventually compute both effect sizes and how they differ.

Thank you all,
Yuhang

PS. James, if I'm reading your website post correctly, it seems that you
define your SMD parameter for your binomial model using one of the group's
SD, but then you estimate that SMD parameter using escalc(measure="SMD")
which uses the pooled SDs of both groups.

On Mon, Apr 3, 2023 at 1:56 AM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Measure "ROM" is not meant to be used for interval scale data. It is
> really meant to be used for data measured on a ratio scale, which has an
> absolute zero point and so the only permissible transformations are
> multiplicative ones (under which the values are again invariant):
>
> x1 <- rlnorm(50, -1, 2)
> x2 <- rlnorm(50, -2, 2)
>
> escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>
> x1 <- x1 * 10
> x2 <- x2 * 10
>
> escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
>                       m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
>
> So, ROM is appropriate for things like height, weight, distance, time, but
> not typically for the types of scales/questionnaires that we construct in
> the social sciences. For example, Likert-type items could be scored from
> 1-7 but we could just as well use 0-6 (and some scales do), which doesn't
> affect SMDs but would lead to different ROM values.
>
> And yes, with correlations, strictly speaking, we also assume linear
> equatability.
>
> In any case, since linear equatability (or with ROMs, multiplicative
> equatability) might not hold, it is useful (if sufficient data is
> available) to examine to what extent there are systematic differences
> between different ways of measuring the same underlying construct.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Yuhang Hu via R-sig-meta-analysis
> >Sent: Monday, 03 April, 2023 3:16
> >To: R meta
> >Cc: Yuhang Hu
> >Subject: Re: [R-meta] SMD Metric
> >
> >Oops, correcting a typo in the second instance of x2:
> >
> >x2 <- 40 + x2 * 3
> >
> >Thank you,
> >Yuhang
> >
> >On Sun, Apr 2, 2023 at 6:02 PM Yuhang Hu <yh342 using nau.edu> wrote:
> >
> >> Thank you, James.
> >>
> >> I think we usually treat items on a test/instrument to be equivalent
> (all
> >> other things equal), if they differ from one another by some additive
> and
> >> multiplicative constants. We can probably extend this inter-item
> >> equivalency concept to equate different tests (all other things equal).
> >>
> >> It seems some effect sizes (e.g., SMD and correlation coef.) take this
> >> into account and 'work best' if such equivalence holds and work less
> well
> >> as this equivalence weakens.
> >>
> >> But other effect sizes seem to be somewhat insensitive to such
> >> equivalency. ROM seems to be one of them. But do you think that it is
> >> desirable for ROM to not give the same estimates for when such an
> >> equivalence, in fact, exists:
> >>
> >> x1 <- rnorm(50, 36, 6)
> >> x2 <- rnorm(50, 33, 6)
> >>
> >> library(metafor)
> >> escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
> >>        m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
> >>
> >> x1 <- 40 + x1 * 3
> >> x2 <- 40 + x2 * 7
> >>
> >> escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
> >>        m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
> >>
> >> Many thanks,
> >> Yuhang
> >>
> >> PS. With a lack of linear equatability, ROM may probably be preferred
> over
> >> SMD. However, I wonder what to replace COR with, when there is a lack of
> >> linear equatability? I would think in that case there is no alternative
> to
> >> COR and we need to resort to adding methodological moderators.
>


-- 
Yuhang Hu (She/Her/Hers)
Ph.D. Student in Applied Linguistics
Department of English
Northern Arizona University

	[[alternative HTML version deleted]]