[R] Benchmark

[Prof Brian D Ripley]

On Tue, 1 Dec 1998, Yoon, Hoon (CICG - NY Program Trading) wrote:

> Hello:
> 
>   I modified benchmark used in
> http://www.informatik.uni-frankfurt.de/~stst/benchspl.txt now the attached
> code will work with R, except for last test for some reason (Sorry ran out
> of time to play with this).

You really do need to say that you did this on Wintel, and give versions of
the packages you used.  As R does not have a BATCH mode that runs on NT
AFAIK, I assume you source-ed the code, which is not a good idea and can be
most misleading due to different memory usage from typing at the command
line.  In short, with benchmarks the details do matter greatly.

>   Immediate problem is that R does not show fraction of seconds. So, that
> kinda makes comparison some what hard. I also looked at memory and CPU usage
> on NT console. R seems to go into hybernation when not focused. The results
> are mixed. I am sure if I add something complicated or/and loops that Splus
> will slow down to crawl.
>   Another interesting thing is that the random generator on Splus is a
> rocket.

No, it isn't!  As RNGs go it is quite slow, but the R one is very slow
(despite the title of the paper `efficient' for which they  never gave any 
evidence).  Lots of timings in a paper of mine written a decade ago:

Ripley, B. D. (1990) Thoughts on pseudorandom number generators. J. Comput.
Appl. Math. 31, 153-163. 

Not that in many real tasks you will notice the difference: there are some
but I would not do those in R.

>   I had many things running on the computer while trying to do this, so I am
> very confident about results.
>   Has anyone done something similar? If so let me know.

These sort of tests do not test the language nor typical uses of it.
They test a few of the built-in routines, thereby mainly testing the
compilers used but also the accuracy/speed compromises used. Thus for
example for an eigendecomposition I do want to know not only the speed but
also the accuracy.

I have run fairly extensive tests of real code (from V&R, for example),
mainly to find problems.  I won't give you extensive sets of numbers, but
re-iterate what I said recently on R-devel: the times are almost always
within a factor of two, and on Solaris S-PLUS 3.4 is usually a little
faster than R 0.63.0. (I am not going to comment on something as
experimental as R on Windows as although I do now have a batch mode running
reliably on Win9x, this is very recent.)

Note that, having tried it, I do NOT believe `if I add something
complicated or/and loops that Splus will slow down to crawl'. Your
prejudices are showing!

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._