[R-sig-hpc] looking for available R benchmarks

[Red Roo]

________________________________
From: xiaoming gu <xiaoming.gu at gmail.com>

Cc: r-sig-hpc at r-project.org
Sent: Wed, April 7, 2010 9:22:58 AM
Subject: Re: [R-sig-hpc] looking for available R benchmarks






----- Original Message ----
>>From: xiaoming gu <xiaoming.gu at gmail.com>
>>To: r-sig-hpc at r-project.org
>>Sent: Tue, April 6, 2010 7:45:05 PM
>>Subject: [R-sig-hpc] looking for available R benchmarks
>
>>Hi, all. I'm a stranger in R and wondering whether there are some available
>>standard benchmarks for R. I would like to know some workloads written in R
>>and designed for the purpose similar to SPEC CPU2006. And each workload has
>>several different input levels. Thanks very much.
>
>
>Xiaoming [[alternative HTML version deleted]]
>>_______________________________________________
>
>It might be helpful if you could clarify the intent of your question.
>
>>Broadly speaking, there are two motivations for benching:
>>a) Comparing the performance of different processors (the dependent variable) running the same R code (independent variable)
>
>>b) Comparing the performance of different R codes (dependent variable) running on the same processor (independent variable)
>
>>Approach (a) is the motivation behind the SPEC benching cartel. As far as I'm aware, the SPEC b/m doesn't contain any R codes but even if it did, their choice of code would not necessarily be the best choice for your purposes. You could, however, apply the same idea using R codes that you know and love. Although this is easy to say, it's not necessarily easy to do correctly. It's very easy to do a lot of work that is meaningless. That's one reason SPEC pre-packages the runtime codes and analysis.
>
>>In approach (b), you compare code performance (e.g., execution times) on a fixed choice of processor, e.g., a machine that has already been purchased. That's what is reported here http://www.revolution-computing.com/products/benchmarks.php  The test rig is fixed as a Dell XPS Studio PC with 2.40GHz Intel Core 2 Quad processors, 6GB system RAM running Windows Server 2008. They then compare the performance of their REvolution R codes with R-2.7.2 from CRAN.
>
>>Although your question would appear to be closer to (a) in intent, it's not clear to me that (b) might not be the better choice. I would need to know more.
>
>>Also, I'm not sure what you mean by "different input levels."
>
>>--Neil Gunther
>
Thanks for all your kindly replies. My intention is closer to (b). I'm trying http://r.research.att.com/benchmarks/ and REvolution R. 

"different input levels" is typically about running time. For example, a very large input makes program run several hours but a small input only takes less than one minute. With this feature, I can study R program performance in a more flexible way.

Xiaoming


>>________________________________________________________________________
>

Case in point for benching approach (b):
Comparative performance of different R implementations for table-lookups on a fixed test platform, viz., elapsed times using system.time() on a Macbook with 2 GHz, and 4 GB RAM.

http://broadcast.oreilly.com/2010/03/lookup-performance-in-r.html


 --njg
________________________________________________________________________
TAKING THE PITH OUT OF PERFORMANCE http://perfdynamics.blogspot.com/
Follow me on Twitter http://twitter.com/DrQz
PERFORMANCE DYNAMICS COMPANY http://www.perfdynamics.com/
Castro Valley, CA 94552