[R] Reg. consensus ranking

[Felix Andrews]

Mallika,

I am not sure exactly what you mean by "consensus approach". One easy
thing you can do is compute the "Pareto front", which is the set of
non-dominated models. A model is dominated or "covered" if another
model exists which is unambiguously better according to the given
scores. So, this method allows you to eliminate uninteresting models
as a first step.

## assume high scores are good
"%covers%" <- function(a, b) all(a >= b) && any(a > b)

> 1:5 %covers% 0:4
[1] TRUE
> 1:5 %covers% 4:0
[1] FALSE

Say you have a matrix with a row for each model and their scores in columns.

foo <- matrix(nrow=1000, ncol=8)
colnames(foo) <- paste("variable", 1:ncol(foo), sep="")
rownames(foo) <- paste("model", 1:nrow(foo), sep="")
foo[] <- rnorm(length(foo))

## compute the set of dominated models (SLOW)
## (for any serious application, write this in C)

dominated <- function(data) {
    apply(data, 1, function(rowi)
        any(apply(data, 1, function(rowj)
            rowj %covers% rowi)))
}

nondom <- !dominated(foo)

> sum(nondom)
[1] 505

So in this case, only about half the cases can be eliminated. But
hopefully your scores will agree more than these random numbers do, so
you will get a bit further. I do think that 20 indicators is probably
too many to get a useful result from a consensus approach, so you
might want to look at subsets of indicators. You should also consider
the uncertainty inherent in the models and indicators when comparing
them.

An extension is to work with the cover matrix, which records which
models are dominated by which others. This defines a graph (as in
graph theory), and you can plot it as a "Hasse diagram" to see
groupings etc. Take the transitive reduction first.

Here's a good reference:

Patil, G.P. and C. Taillie (2004), Multiple indicators, partially
ordered sets, and linear extensions: Multi-criterion ranking and
prioritization, Environmental and Ecological Statistics, 11, 199-228.

and maybe <cough>

Andrews, F. (2005). Representing Uncertainty in Ranking by Single or
Multiple Indicators. In Zerger, A. and Argent, R.M. (eds) MODSIM 2005
International Congress on Modelling and Simulation. Modelling and
Simulation Society of Australia and New Zealand, December 2005, pp.
2456-2462. ISBN: 0-9758400-2-9.
http://www.mssanz.org.au/modsim05/papers/andrews.pdf



On Mon, Apr 21, 2008 at 8:17 AM, Mallika Veeramalai
<mallikav at burnham.org> wrote:
>
>  Dear All,
>
>  I have a list of models(1000) which have variable scores from 20 different method.  I would like to rank models using consensus approach based on high scores from different methods.Is there any function available in R for this purpose?  I will appreciate any pointers in this regard.
>
>
>  Thank you very much in Advance,
>  Mallika
>
>
>  *~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*
>  Mallika Veeramalai, PhD,
>  Postdoctoral Associate,
>  Bioinformatics & Systems Biology,
>  Prof. Adam Godzik Lab,
>  Burnham Institute for Medical Research,
>  La Jolla, San Diego, CA 92037, US.
>
>  phone : +1 858 646 3100 ext: 3627 (work)
>  Fax   : +1 858 795 5249
>  Web   : http://bioinformatics.burnham.org/~mallika/
>  Email : mallikav at burnham.org
>         kaaviyam at gmail.com
>  *~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*
>
> ______________________________________________
>  R-help at r-project.org mailing list
>  https://stat.ethz.ch/mailman/listinfo/r-help
>  PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>  and provide commented, minimal, self-contained, reproducible code.
>
>



-- 
Felix Andrews / $B0BJ!N)(B
PhD candidate
Integrated Catchment Assessment and Management Centre
The Fenner School of Environment and Society
The Australian National University (Building 48A), ACT 0200
Beijing Bag, Locked Bag 40, Kingston ACT 2604
http://www.neurofractal.org/felix/
3358 543D AAC6 22C2 D336 80D9 360B 72DD 3E4C F5D8