[R-sig-hpc] looking for available R benchmarks

[Red Roo]

----- Original Message ----
From: xiaoming gu <xiaoming.gu at gmail.com>
To: r-sig-hpc at r-project.org
Sent: Tue, April 6, 2010 7:45:05 PM
Subject: [R-sig-hpc] looking for available R benchmarks

Hi, all. I'm a stranger in R and wondering whether there are some available
standard benchmarks for R. I would like to know some workloads written in R
and designed for the purpose similar to SPEC CPU2006. And each workload has
several different input levels. Thanks very much.

Xiaoming [[alternative HTML version deleted]]
_______________________________________________

It might be helpful if you could clarify the intent of your question. 

Broadly speaking, there are two motivations for benching:
a) Comparing the performance of different processors (the dependent variable) running the same R code (independent variable) 

b) Comparing the performance of different R codes (dependent variable) running on the same processor (independent variable)

Approach (a) is the motivation behind the SPEC benching cartel. As far as I'm aware, the SPEC b/m doesn't contain any R codes but even if it did, their choice of code would not necessarily be the best choice for your purposes. You could, however, apply the same idea using R codes that you know and love. Although this is easy to say, it's not necessarily easy to do correctly. It's very easy to do a lot of work that is meaningless. That's one reason SPEC pre-packages the runtime codes and analysis.

In approach (b), you compare code performance (e.g., execution times) on a fixed choice of processor, e.g., a machine that has already been purchased. That's what is reported here http://www.revolution-computing.com/products/benchmarks.php  The test rig is fixed as a Dell XPS Studio PC with 2.40GHz Intel Core 2 Quad processors, 6GB system RAM running Windows Server 2008. They then compare the performance of their REvolution R codes with R-2.7.2 from CRAN.

Although your question would appear to be closer to (a) in intent, it's not clear to me that (b) might not be the better choice. I would need to know more.

Also, I'm not sure what you mean by "different input levels."

--Neil Gunther
________________________________________________________________________
TAKING THE PITH OUT OF PERFORMANCE http://perfdynamics.blogspot.com/
Follow me on Twitter http://twitter.com/DrQz
PERFORMANCE DYNAMICS COMPANY http://www.perfdynamics.com/
Castro Valley, CA 94552