[R-sig-ME] Help with mixed model design

[Chad Newbolt]

All,



I'm having some difficulty envisioning how to address a question with mixed model and thought might find some assistance here.  I have a dataset generated from a survey where individual were asked to view a set of images and classify animals in these images according to sex/age groups.  We wanted to investigate the effects of observer experience, profession, etc., and factors related to the image itself (day/night, animal group, etc.) on how accurately observers could classify animals.  Responses in our survey could be either correct/incorrect (we knew truth in our test images) which could be input as a simple binary response; however, viewers also had the option of responding  "Unknown" for images they did not feel comfortable answering due to image quality, etc.  This was done to mimic what is done in practice in our field of work but was a real pain in our current efforts.  Since "Unknown" responses in our particular application are technically neither correct nor incorrect (but do influence census results in practice), I decided to split my analysis into two models: 1) A correct/incorrect response model (I removed the Unknown responses in this data set), and 2) An Unknown response model (I created a new binary variable for Unknown/Any other response).  These models answer two very different questions which we believe are both of value to us.



My difficulty now concerns the Incorrect responses.  I would like to find a way to determine probabilities of the kinds of error that we see in our data.  For example, images that contained male animals that were incorrectly answered, how likely were those responses female vs. young?  This information would be valuable in predicting how the errors that we observed might be influencing population surveys.



My thought is to first subset the data, maintaining only incorrect responses.  I assume I could then create a new binary factor where one level of observer responses (Male, Female, Young) is 1 and all others are 0.  I could then model this binary response with a fixed effect for the answer I know to be true (also Male, Female, Young) along with my random effects.  This would be repeated three times for each kind of incorrect response.  Something like:



Response"Male 1 or 0" ~  True Answer + (1| Observer ID) + (1|Question)

Response"Female 1 or 0" ~  True Answer + (1| Observer ID) + (1|Question)

Response"Young 1 or 0" ~  True Answer + (1| Observer ID) + (1|Question)



Does that make any sense?  Any better ideas?  It's probably very simple but I've really struggled with this one.



Thanks

	[[alternative HTML version deleted]]