[R-sig-hpc] Parallel operations
Drew Schmidt
schmidt at math.utk.edu
Tue Apr 29 18:17:45 CEST 2014
Hi,
Both the frameworks you mentioned perform operations in memory (like
R). And because this is R, you should be holding out space for 2-4
copies in ram for results depending on the operation. It sounds like
you either need more ram or out of core methods. I believe RevoScaleR
does the latter.
HIPLARb replaces R matrix algebra functionality, while HIPLARM replaces
Matrix (the package) functionality. For the moment, pbdR matrix
operations are exclusively dense as well. Additionally, pbdR is meant
for distributed environments. While you can achieve speedups with pbdR
on smaller scale hardware/problems, MPI overhead is nontrivial. For a
small shared memory node (commodity hardware), I would expect PLASMA to
outperform ScaLAPACK. This picture gets a bit more complicated as
hardware gets bigger and things like Intel MKL get thrown into the mix.
Lastly, if you want to compute the actual elements of an inverse, it
doesn't really matter how sparse your matrix is. In general, the
inverse of a sparse matrix is dense. I would make sure that you really
do need to invert the matrix before heading down that path.
--
Drew Schmidt
National Institute for Computational Sciences
University of Tennessee, USA
http://r-pbd.org/
On 04/29/2014 11:08 AM, Bill Smith wrote:
> I have a Windows box set up with two 64 bit Intel Xeon X5680 3.33 GHz processors (6 cores each) and 12 GB of RAM. I want to use it to perform matrix operations (multiplication, inversion) on some data set sets that range from 5-50 GB. I looked around on the web, including the CRAN HPC, to see what was available. A couple of things came up, pbdr (http://r-pbd.org/) and HIPLAR (http://hiplar.org/hiplar-b.html). Which of these is the best for my application? It's not even clear HIPLAR will work in a Windows environment. If I go the HIPLAR route, which of the two programs, HIPLARb or HIPLARM, is preferable? It's not clear to me the difference in the two programs. My matrices have a lot of zeros, but the HIPLAR software requires PLASMA/MAGMA which seems to only work with dense matrices. Can someone help me sort this out?
>
> A related question concerns the availability of an R package to perform stepwise logistic regression (yes, I know this is an unpardonable sin-I'm replicating someone else's analysis) in a parallel environment. I believe this has been implemented in Revolution R, but I'm not sure if anywhere else. Any guidance appreciated.
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
More information about the R-sig-hpc
mailing list