[R-sig-hpc] What to Experiment With?

[Whit Armstrong]

Save your money.

EC2...
rzmq...
deathstar...

It's as simple as this:
cl <- startCluster(ami="ami-9d5f93f4",key="maher-ave",instance.count=2,instance.type="c1.xlarge")
print("starting sim.")
run.time <- system.time(ans <-
zmq.cluster.lapply(cluster=cl$instances[,"dnsName"],as.list(1:1e3),estimatePi))
print("sim completed.")
res <- terminateCluster(cl)

Your functions and your data go across the wire, and the results come
back to your machine.  You don't have to log into any cloud instances
to execute this, it runs from your desktop...

There is no need to buy big machines anymore unless you have highly
sensitive data (that you don't want going over the wire) or you need
co-location.

<shameless plug>
I'll be talking about these tools at R/Finance (come to the conference!)
http://www.rinfinance.com/agenda/
</shameless plug>


-Whit



On Fri, Apr 20, 2012 at 6:16 PM, ivo welch <ivowel at gmail.com> wrote:
> Dear R HPC experts:
>
> I have about $5,000 to spend on building fast computer hardware to run
> our problems.  if it works well, I may be able to scrounge up another
> $10k/year to scale it up.  I do not have the resources to program very
> complex algorithms, administer a full cluster, etc.  (the effective
> programmer's rate here is about $50/hour and up, and I have severe
> restrictions against hiring outsiders.)  the programs basically have
> to work with minimum special tweaking.
>
> There are no real-time needs.  Typically, I operate on historical CRSP
> and Compustat data, which are about 1-5GB (depending on subset).  most
> of what I am doing involves linear regressions.  I often need to
> calculate Newey-West/Hansen-Hodrick/White adjusted standard errors,
> and I often do need to sort and rank, calculate means and covariances.
>  these are not highly sophisticated stats, but it entails lots of it.
> most of what I do is embarrassingly parallel.
>
> Now, I think in the $5k price range, I have a couple of options.
> Roughly, the landscape seems to be:
>
> * 1 dual-socket xeon i7 computers.
> * 5 (desktop) i7 computers, networked (socket snow?).
> * 1 i7 computer, with 1 nvidia Tesla card
> * 1 i7 computers with 2-3 commodity graphics cards
>     --- apparently, nvidia cripples the DP performance of its gamer
> cards, so AMD should be a *lot* faster
>     at the same price, but I only see the lm() routine in
> nvidia-specific gputools.  then again, for Newey-West,
>     I may have to resort to my own calculations, anyway.  is there
> newey-west OLS code for AMD GPUs?
>
> I would presume that an internal PCI bus is a lot faster than an
> ethernet network, and a GPU could be faster than a CPU, but a GPU is
> also less flexible.  Sigh...not sure.
> what should I try?
>
> /iaw
>
>
> ----
> Ivo Welch (ivo.welch at gmail.com)
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc