Thanks Steve,

I'm halfway there with:

foo <- cbind(foo, range_group=cut(foo$score, breaks=c(.9, .8, .7, .6, 
.5, .4, .3, .2, .1)))
with(foo, tapply(score, list(range_group), mean))


This works, but I only get one of the 3 columns I need, mean(score).

I'm not sure how to get the other two.

It really is the result of a few columns.  For example:
a <- count of entries in this "range"
b <- number of entries in this "range" labeled true
c < b / a   (What percentage of entries in this range are really true


My guess is that there is a way to add more columns to the tapply, but 
I'm not sure where/how.

-N

On 8/5/09 11:22 AM, Steve Lianoglou wrote:
> Hi,
>
> On Aug 5, 2009, at 2:11 PM, Noah Silverman wrote:
>
>> Hello,
>>
>> I asked this as part of a previous message, but never really figured 
>> out a usable solution.  So this is a second attempt.
>>
>> I have an process containing an SVM.  The end result is the 
>> probability that the class is true.  That result is added back to the 
>> original data.
>>
>> So I wind up with a data.frame that looks like this
>>
>> label,v1,v2,v3,prob_true
>>
>> What I want to do is measure how accurate my model is for each range 
>> of probability.  (I've seen this done is a few published papers and 
>> found it a very useful way to visualize things.)
>>
>> My hope/guess is that there is some kind of package for R that does 
>> this since it should be a common need.
>>
>> Here is an example of what I'd like to be able to generate:
>>
>> range        number of items        mean(probability)   true_accuracy
>> 100-90%        20                            .924                    .90
>> 90-80%          50                            .825                    
>> .84
>> 80-70%          214                          .75                      
>> .71
>> etc...
>>
>> range is the range of predicted values by the SVM
>> mean(probability) is the mean of the PREDICTED probability of items 
>> in that range
>> true_accuracy is the mean of the ACTUAL probability of items in that 
>> range.
>>
>> In English I would explain it as, "Of the data where our SVM 
>> predicted a true probability of 70-80%, the data was actually 71% true."
>>
>> It might be really  helpful to be able to graph this somehow.  
>> (Again, There must be some package in R for this??)
>> With mean(predicted_probability) on one axis and 
>> mean(true_probability) on the other axis.
>>
>> Any thoughts, comments, ideas, etc. would be appreciated!
>
> Take a look at the cut function, and the code in the examples of ?cut 
> (eg, take a look at the output when combined w/ table(cut(..)) ).
>
> Sending in your own vector for the ``breaks`` param inorder to bin as 
> you like should get you 90% of the way to building the table you're 
> after.
>
> -steve
>
> -- 
> Steve Lianoglou
> Graduate Student: Computational Systems Biology
>   |  Memorial Sloan-Kettering Cancer Center
>   |  Weill Medical College of Cornell University
> Contact Info: http://cbio.mskcc.org/~lianos/contact
>

	[[alternative HTML version deleted]]


