Hi Ramon,

I'm not sure that I fully understand the details of what you want to accomplish. But I do want to ask: you jump right into your email assuming that of course you want to model precision and recall, but what about modelling the data directly (i.e., individual classification decisions) rather than summaries of the data? Then you could work backward (forward?) from the model results to compute what the implied precision and recall would be.

If you decided that modelling the data directly would work for your purposes, then one way of doing this would be to regress classification decisions ("P" or "N") on actual classifications ("P" or "N"). If this is done in a probit model, it is equivalent to the equal-variance signal detection model studied at length in psychology, with the intercept being the "criterion" in signal detection language (denoted c), and the slope being "sensitivity" (denoted d' or d-prime). It should definitely be possible to compute precision and recall from c and d'. This might be simpler with a logit rather than probit link function.

Let me know if I have misunderstood what you are trying to accomplish.

Jake

> From: rdiaz02@gmail.com
> To: r-sig-mixed-models@r-project.org
> Date: Tue, 11 Mar 2014 11:48:57 +0100
> CC: ramon.diaz@iib.uam.es
> Subject: [R-sig-ME] Modeling precision and recall with GLMMs
> 
> Dear All,
> 
> I am examining the performance of a couple of classification-like methods
> under different scenarios. Two of the metrics I am using are precision and
> recall (TP/(TP + FP) and TP/(TP + FN), where TP, FP, and FN are "true
> positives", "false positives", and "false negatives" in a simple two-way
> confusion matrix). Some of the combinations of methods have been used on
> exactly the same data sets. So it is easy to set up a binomial model (or
> multinomial2 if using MCMCglmm) such as
> 
> 
> cbind(TP, FP) ~ fixed effects + (1|dataset) 
> 
> 
> 
> However, the left hand side sounds questionable, specially with precision:
> the expression TP/(TP + FP) has, in the denominator, a (TP + FP) [the
> number of results returned, or retrieved instances, etc] that, itself, can
> be highly method-dependent (i.e., affected by the fixed effects). So rather
> than a true proportion, this seems more like a ratio, where each of TP and
> FP have their own variance, a covariance, etc, and thus the error
> distribution is a mess (not the tidy thing of a binomial).
> 
> 
> I've looked around in the literature and have not found much (maybe the
> problem are my searching skills :-). Most people use rankings of methods,
> not directly modeling precision or recall in the left-hand side of a
> (generalized) linear model. A couple of papers use a linear model on the
> log-transformed response (which I think is even worse than the above
> binomial model, specially with lots of 0s or 1s). Some other people use a
> single measure, such as the F-measure or Matthews correlation coefficient,
> and I am using something similar too, but I specifically wanted to also
> model precision and recall.
> 
> 
> An option would be a multi-response model with MCMCglmm, but I am not sure
> if this is appropriate either (dependence of the sum of FP and TP on the
> fixed effects).
> 
> 
> Best,
> 
> 
> R.
> 
> -- 
> Ramon Diaz-Uriarte
> Department of Biochemistry, Lab B-25
> Facultad de Medicina 
> Universidad Autónoma de Madrid 
> Arzobispo Morcillo, 4
> 28029 Madrid
> Spain
> 
> Phone: +34-91-497-2412
> 
> Email: rdiaz02@gmail.com
>        ramon.diaz@iib.uam.es
> 
> http://ligarto.org/rdiaz
> 
> _______________________________________________
> R-sig-mixed-models@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
 		 	   		  
	[[alternative HTML version deleted]]

