[R-meta] Meta analysis of probability values (Ruscio's A/CLES)

[ian hussey]

Hi James,

Thanks for these thoughts.

As an aside, I think Ruscios A would be more popular if it didn’t go by so
many names - you’ve added at least one new one to those I know.

I have previously used your last suggestion of a logit transformation, but
as you say many native values are 1.0 and this presents a ceiling effect.
I’ll look into using a glm with logit link, thanks.

Best
Ian
On Thu 13 Sep 2018 at 20:14 James Pustejovsky <jepusto using gmail.com> wrote:

> Ian,
>
> Ruscio's A is equivalent to the non-overlap of all pairs (Parker &
> Vannest, 2009) from the single-case research literature, and it has various
> other names from the medical statistics literature (stochastic superiority
> index, ordinal dominance, etc.). In the latter context, there's a paper by
> Ryu & Agresti (2008) that looks a little bit at a fixed-effect
> meta-analysis model for A (or actually for a generalization of A). Their
> idea is to use a generalized linear model with a logit link. This could be
> extended to a generalized linear mixed model to capture true heterogeneity
> in the effect size parameters.
>
> A simpler thing to do would be to just use a logit transformation of A and
> work out (or bootstrap) the standard error of the transformed ES estimate
> (see Mee, 1990 for helpful theory here). A problem with this approach is
> that in practice the estimates are very often at ceiling of 1 and logit(1)
> = infinity.
>
> James
>
>
> Mee, W. (1990). Confidence intervals for probabilities and tolerance
> regions based on a generalization of the Mann-Whitney statistic. Journal of
> the American Statistical Association, 85(411), 793–800.
> https://doi.org/10.1080/01621459.1990.10474942
>
> Parker, R. I., & Vannest, K. J. (2009). An improved effect size for
> single-case research: Nonoverlap of all pairs. Behavior Therapy, 40(4),
> 357–67. https://doi.org/10.1016/j.beth.2008.10.006
>
> Ryu, E., & Agresti, A. (2008). Modeling and inference for an ordinal
> effect size measure. Statistics in Medicine, 27(10), 1703–1717.
> https://doi.org/10.1002/sim.3079
>
> On Thu, Sep 13, 2018 at 7:25 AM ian hussey <ian.hussey using ugent.be> wrote:
>
>> Hi all,
>>
>> I'm writing an R package to do robust analysis of A-B Single Case
>> Experimental Design data (https://github.com/ianhussey/SCED). I'm
>> estimating Ruscio's A effect size for each participant. These are
>> probability values and are akin to the Common Language Effect Size. I want
>> to add a RE meta analysis across participants to find the meta analytic
>> Ruscio's A, its CIs, and CRs.
>>
>> I haven't managed to find any literature on the meta analysis of
>> probability values. I've read metafor's documentation (e.g.,
>> https://rdrr.io/cran/metafor/man/escalc.html) but haven't found specific
>> accommodations for probability values. I have considered analysing them
>> using the package defaults for ES, but this produces a forest plot with
>> max
>> CI values above probability of 1.0, and measures of heterogeneity seem to
>> return extreme results. It also does not accommodate the fact that CIs on
>> probability values are frequently asymmetrical. Does anyone have any
>> advice?
>>
>> Here is a minimal reproducible example in case it's useful:
>>
>> https://github.com/ianhussey/SCED/blob/master/vignettes/meta_analysis_development.Rmd
>>
>> Thanks,
>> Ian
>>
>> --
>> Ian Hussey
>> Postdoctoral research fellow
>> Department of Experimental-Clinical and Health Psychology
>> Ghent University
>>
>> researchgate.net/profile/ian_hussey
>> mmmdata.io
>> twitter.com/ianhussey
>> osf.io/3kzh8
>> github.com/ianhussey
>> --
>> Ian Hussey
>> Postdoctoral research fellow
>> Department of Experimental-Clinical and Health Psychology
>> Ghent University
>>
>> researchgate.net/profile/ian_hussey
>> mmmdata.io
>> twitter.com/ianhussey
>> osf.io/3kzh8
>> github.com/ianhussey
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
> --
Ian Hussey
Postdoctoral research fellow
Department of Experimental-Clinical and Health Psychology
Ghent University

researchgate.net/profile/ian_hussey
mmmdata.io
twitter.com/ianhussey
osf.io/3kzh8
github.com/ianhussey

	[[alternative HTML version deleted]]