[R-sig-ME] Analysis of signal detection data

[Thompson,Paul]

Mike:

This situation is well described and most appropriately analyzed by the use of GEE methods.  Using GEE methods, you may examine data observed multiple Level 1 times for a given Level 2 unit (that is, multi-level data), but with a binomial (hit-miss, yes-no), Poisson (count), ordered categorical (ordinal), or several other distributions.  In these methods, you use linear models methods of least-squares estimation to obtain coefficients, but then use an appropriate error term to model the errors.

You may think of this as logistic regression with repeated measures if you wish since logistic regression involving yes-no data and generalized linear models involving binomial link function are the same model. And ROC curve analysis is performed more efficiently, and in a manner incorporating multiple covariates, using logistic regression. 

In short, your suggestion is a good one, and is quite well encompassed (but possibly not completely) using existing methods.

Software for these methods includes PROC GENMOD and LOGISTIC in SAS, MPlus for multivariate structural equation modeling of this logistic binomially distributed data, and a number of other programs as well.



-----Original Message-----
From: r-sig-mixed-models-bounces at r-project.org on behalf of Mike Lawrence
Sent: Sun 11/14/2010 9:58 PM
To: r-sig-mixed-models at r-project.org
Subject: [R-sig-ME] Analysis of signal detection data
 
Hi folks,

Yet another query on whether traditional stats employed in psychology
might be improved by mixed effects modelling...

Consider a radiologist looking at a CT scan and attempting to make the
binary diagnosis of cancer/no cancer. Signal detection theory suggests
that the normalized difference between the radiologist's hit rate and
false alarm rate provides a metric of the radiologist's discrimination
skill (d'). That is:

d' = qnorm(hit_rate) - qnorm(FA_rate)

Now, if we wanted to see if discrimination skill was improved by some
intervention, we might recruit a bunch of radiologists and measure
their d' both before and after the intervention. That is, both before
the intervention, each radiologist would be presented with a number of
"trials" where they review CT scans, mark them as cancer/no cancer,
and we experimentalists score each diagnosis as a hit, miss, false
alarm, or correct rejection.

Presented with data like this, most psychologists would compute a d'
score for each radiologist both before and after the intervention,
then submit the d' scores to a repeated-measures ANOVA, which assumes
gaussian error. However, hit and false alarm rates should yield
binomially distributed error distributions, and monte carlo
experimentation in R leads me to believe that in cases where only a
moderate number of CT scans are reviewed per session (say, 10-20), d'
may be expected to be considerably non-gaussian.

I know mixed effects modelling can handle binomially distributed
error, but is there any way to handle this sort of signal detection
data? My first thought is that glmmer with 4 categories corresponding
to the hit, miss, false alarm, and correction categorization of
responses, but I don't immediately see how this would properly connect
the hit-vs-miss data to reflect a hit rate and the
false-alarm-vs-correct-rejection data to reflect a FA rate.

Thoughts?

Mike

--
Mike Lawrence
Graduate Student
Department of Psychology
Dalhousie University

Looking to arrange a meeting? Check my public calendar:
http://tr.im/mikes_public_calendar

~ Certainty is folly... I think. ~

_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models


-----------------------------------------------------------------------
Please note that My Email Address Has Changed!
Please begin using the address in the "From" line above, immediately.
Soon, email sent to my old address will no longer be delivered to me.

Confidentiality Notice: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
privileged and confidential information.  Any unauthorized review, use,
disclosure or distribution is prohibited.  If you are not the intended
recipient, please contact the sender by reply e-mail and destroy
all copies of the original message.