[R-sig-hpc] Parallel operations

[Drew Schmidt]

Hi,

Both the frameworks you mentioned perform operations in memory (like 
R).  And because this is R, you should be holding out space for 2-4 
copies in ram for results depending on the operation.  It sounds like 
you either need more ram or out of core methods.  I believe RevoScaleR 
does the latter.

HIPLARb replaces R matrix algebra functionality, while HIPLARM replaces 
Matrix (the package) functionality.  For the moment, pbdR matrix 
operations are exclusively dense as well.  Additionally, pbdR is meant 
for distributed environments.  While you can achieve speedups with pbdR 
on smaller scale hardware/problems, MPI overhead is nontrivial.  For a 
small shared memory node (commodity hardware), I would expect PLASMA to 
outperform ScaLAPACK.  This picture gets a bit more complicated as 
hardware gets bigger and things like Intel MKL get thrown into the mix.

Lastly, if you want to compute the actual elements of an inverse, it 
doesn't really matter how sparse your matrix is.  In general, the 
inverse of a sparse matrix is dense.  I would make sure that you really 
do need to invert the matrix before heading down that path.

--
Drew Schmidt
National Institute for Computational Sciences
University of Tennessee, USA
http://r-pbd.org/

On 04/29/2014 11:08 AM, Bill Smith wrote:
> I have a Windows box set up with two 64 bit Intel Xeon X5680 3.33 GHz processors (6 cores each) and 12 GB of RAM. I want to use it to perform matrix operations (multiplication, inversion) on some data set sets that range from 5-50 GB.  I looked around on the web, including the CRAN HPC, to see what was available. A couple of things came up, pbdr (http://r-pbd.org/) and HIPLAR (http://hiplar.org/hiplar-b.html).  Which of these is the best for my application?  It's not even clear HIPLAR will work in a Windows environment.  If I go the HIPLAR route, which of the two programs, HIPLARb or HIPLARM, is preferable?  It's not clear to me the difference in the two programs.  My matrices have a lot of zeros, but the HIPLAR software requires PLASMA/MAGMA which seems to only work with dense matrices.  Can someone help me sort this out?
>   
> A related question concerns the availability of an R package to perform stepwise logistic regression (yes, I know this is an unpardonable sin-I'm replicating someone else's analysis) in a parallel environment.  I believe this has been implemented in Revolution R, but I'm not sure if anywhere else.  Any guidance appreciated.
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc