[R-meta] Meta analysis of probability values (Ruscio's A/CLES)

[James Pustejovsky]

Ian,

Ruscio's A is equivalent to the non-overlap of all pairs (Parker & Vannest,
2009) from the single-case research literature, and it has various other
names from the medical statistics literature (stochastic superiority index,
ordinal dominance, etc.). In the latter context, there's a paper by Ryu &
Agresti (2008) that looks a little bit at a fixed-effect meta-analysis
model for A (or actually for a generalization of A). Their idea is to use a
generalized linear model with a logit link. This could be extended to a
generalized linear mixed model to capture true heterogeneity in the effect
size parameters.

A simpler thing to do would be to just use a logit transformation of A and
work out (or bootstrap) the standard error of the transformed ES estimate
(see Mee, 1990 for helpful theory here). A problem with this approach is
that in practice the estimates are very often at ceiling of 1 and logit(1)
= infinity.

James


Mee, W. (1990). Confidence intervals for probabilities and tolerance
regions based on a generalization of the Mann-Whitney statistic. Journal of
the American Statistical Association, 85(411), 793–800.
https://doi.org/10.1080/01621459.1990.10474942

Parker, R. I., & Vannest, K. J. (2009). An improved effect size for
single-case research: Nonoverlap of all pairs. Behavior Therapy, 40(4),
357–67. https://doi.org/10.1016/j.beth.2008.10.006

Ryu, E., & Agresti, A. (2008). Modeling and inference for an ordinal effect
size measure. Statistics in Medicine, 27(10), 1703–1717.
https://doi.org/10.1002/sim.3079

On Thu, Sep 13, 2018 at 7:25 AM ian hussey <ian.hussey using ugent.be> wrote:

> Hi all,
>
> I'm writing an R package to do robust analysis of A-B Single Case
> Experimental Design data (https://github.com/ianhussey/SCED). I'm
> estimating Ruscio's A effect size for each participant. These are
> probability values and are akin to the Common Language Effect Size. I want
> to add a RE meta analysis across participants to find the meta analytic
> Ruscio's A, its CIs, and CRs.
>
> I haven't managed to find any literature on the meta analysis of
> probability values. I've read metafor's documentation (e.g.,
> https://rdrr.io/cran/metafor/man/escalc.html) but haven't found specific
> accommodations for probability values. I have considered analysing them
> using the package defaults for ES, but this produces a forest plot with max
> CI values above probability of 1.0, and measures of heterogeneity seem to
> return extreme results. It also does not accommodate the fact that CIs on
> probability values are frequently asymmetrical. Does anyone have any
> advice?
>
> Here is a minimal reproducible example in case it's useful:
>
> https://github.com/ianhussey/SCED/blob/master/vignettes/meta_analysis_development.Rmd
>
> Thanks,
> Ian
>
> --
> Ian Hussey
> Postdoctoral research fellow
> Department of Experimental-Clinical and Health Psychology
> Ghent University
>
> researchgate.net/profile/ian_hussey
> mmmdata.io
> twitter.com/ianhussey
> osf.io/3kzh8
> github.com/ianhussey
> --
> Ian Hussey
> Postdoctoral research fellow
> Department of Experimental-Clinical and Health Psychology
> Ghent University
>
> researchgate.net/profile/ian_hussey
> mmmdata.io
> twitter.com/ianhussey
> osf.io/3kzh8
> github.com/ianhussey
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]