[R] Proportion test in three-chices experiment

[Jonathan Baron]

I suspect that there are more direct ways to do this test, but it 
is unclear to me just what the issue is.  For example, if there
are many subjects and very few stimuli for each, you might want
to get some sort of measure of ability for each subject (many
possibilities here, then test the measure across subjects with a
t test.  The measure must be chosen so that you can specify a
null hypothesis.  It must be directional.

If you have a few subjects and many trials per subject, then you
could do a significance test for each subject.
You want a directional test, because you have a specific
hypothesis, namely, that the correct answer will occur more often 
than predicted from the marginal frequencies in the 3x3 table.
(I assume it is a 3x3 table with stimuli as rows and responses ad 
columns, and you want to show that the diagonal cells are higher
than predicted.) One possibility is kappa, which is in the vcd
package, and also in psy and concord, in somewhat different
forms.

Usually in this sort of experiment, though, there isn't much of
an issue about whether subjects are transmitting information at
all.  Rather the issue is testing alternative models of what they 
are doing.

Jon

On 07/16/05 06:33, Spencer Graves wrote:
> 	  Have you considered "BTm" in library(BradleyTerry)?  Consider the
> following example:
> 
>  > cond1 <- data.frame(winner=rep(LETTERS[1:3], e=2),
> +           loser=c("B","C","A","C","A","B"),
> +           Freq=1:6)
>  > cond2 <- data.frame(winner=rep(LETTERS[1:3], e=2),
> +           loser=c("B","C","A","C","A","B"),
> +           Freq=6:1)
>  > fit1 <- BTm(cond1~..)
>  > fit2 <- BTm(cond2~..)
>  > fit12 <- BTm(rbind(cond1, cond2)~..)
>  > Dev12 <- (fit1$deviance+fit2$deviance
> +           -fit12$deviance)
>  > pchisq(Dev12, 2, lower=FALSE)
> [1] 0.8660497
> 
> 	  This says the difference between the two data sets, cond1 and cond2,
> are not statistically significant.
> 
> 	  Do you present each subject with onely one pair?  If yes, then this
> model is appropriate.  If no, then the multiple judgments by the same
> subject are not statistically independent, as assumed by this model.
> However, if you don't get statistical significance via this kind of
> computation, it's unlikely that a better model would give you
> statistical significance.  If you get a p value of, say, 0.04, then the
> difference is probably NOT statistically significant.
> 
> 	  The p value you get here would be an upper bound.  You could get a
> lower bound by using only one of the three pairs presented to each
> subject selected at random.  If that p value were statistically
> significant, then I think it is safe to say that your two sets of
> conditions are significantly different.  For any value in between, it
> would depend on how independent the three choices by the same subject.
> You might, for example, delete one of the three pairs at random and use
> the result of that comparison.
> 
> 	  There are doubtless better techniques, but I'm not familiar with
> them.  Perhaps someone else will reply to my reply.
> 
> 	  spencer graves
> 
> Rafael Laboissiere wrote:
> 
> > Hi,
> >
> > I wish to analyze with R the results of a perception experiment in which
> > subjects had to recognize each stimulus among three choices (this was a
> > forced-choice design).  The experiment runs under two different
> > conditions and the data is like the following:
> >
> >    N1 : count of trials in condition 1
> >    p11, p12, p13: proportions of choices 1, 2, and 3 in condition 1
> >
> >    N2 : count of trials in condition 2
> >    p21, p22, p23: proportions of choices 1, 2, and 3 in condition 2
> >
> > How can I test whether the triple (p11,p12,p13) is different from the
> > triple (p21,p22,p23)?  Clearly, prop.test does not help me here, because
> > it relates to two-choices tests.
> >
> > I apologize if the answer is trivial, but I am relatively new to R and
> > could not find any pointers in the FAQ or in the mailing list archives.
> >
> > Thanks in advance for any help,
> >
> 
> --
> Spencer Graves, PhD
> Senior Development Engineer
> PDF Solutions, Inc.
> 333 West San Carlos Street Suite 700
> San Jose, CA 95110, USA
> 
> spencer.graves at pdf.com
> www.pdf.com <http://www.pdf.com>
> Tel:  408-938-4420
> Fax: 408-280-7915
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html

-- 
Jonathan Baron, Professor of Psychology, University of Pennsylvania
Home page: http://www.sas.upenn.edu/~baron