[R] Proportion test in three-chices experiment

[Rafael Laboissiere]

* Jonathan Baron <baron at psych.upenn.edu> [2005-07-16 11:49]:

> I suspect that there are more direct ways to do this test, but it 
> is unclear to me just what the issue is.  For example, if there
> are many subjects and very few stimuli for each, you might want
> to get some sort of measure of ability for each subject (many
> possibilities here, then test the measure across subjects with a
> t test.  The measure must be chosen so that you can specify a
> null hypothesis.  It must be directional.
> 
> If you have a few subjects and many trials per subject, then you
> could do a significance test for each subject.
> You want a directional test, because you have a specific
> hypothesis, namely, that the correct answer will occur more often 
> than predicted from the marginal frequencies in the 3x3 table.
> (I assume it is a 3x3 table with stimuli as rows and responses ad 
> columns, and you want to show that the diagonal cells are higher
> than predicted.) One possibility is kappa, which is in the vcd
> package, and also in psy and concord, in somewhat different
> forms.

Thanks for your reply, Jonathan.  Thanks also to Spencer, who suggested
using the BTm function.  I realize that my description of both the
experiment and the involved issue was not clear.  Let me try again:

My subjects do a recognition task where I present stimuli belonging to
three different classes (let us say A, B, and C).  There are many of
them.  Subjects are asked to recognize each stimulus as belonging to one
of the three classes (forced-choice design).  This is done under two
different conditions (say conditions 1 and 2).  I end up with matrices of
counts like this (in R notation):

# under condition 1
c1 <- t (matrix (c (c1AA, c1AB, c1AC, 
                    c1BA, c1BB, c1BC, 
		    c1CA, c1CB, c1CC), nc = 3))
# under condition 2
c2 <- t (matrix (c (c2AA, c2AB, c2AC, 
                    c2BA, c2BB, c2BC, 
		    c2CA, c2CB, c2CC), nc = 3))

where "cijk" is the number of times the subject gave answer k when
presented with a stimulus of class j, under condition i.

The issue is to test whether subjects perform better (in the sense of a
higher recognition score) in condition 1 compared with condition 2.  My
first idea was to test the global recognition rate, which could be
computed as:

# under condition 1
r1 <- sum (diag (c1)) / sum (c1)
# under condition 2
r2 <- sum (diag (c2)) / sum (c2)

The null hypothesis is that r1 is not different from r2. I guess that I
could test it with the chisq.test function, like this:

p1 <- sum (diag (c1))
q1 <- sum (c1) - p1
p2 <- sum (diag (c2))
q2 <- sum (c2) - p2
chisq.test (matrix (c(p1, q1, p2, q2), nc = 2))

What do you think?

I also thought about testing the triples like [c1AA, c1AB, c1AC] against
[c2AA, c2AB, c2AC], hence my original question.

-- 
Rafael