Dear Berend,
thanks to everyone who responded. I should point that the data matrix is not sparse. Yes, the irlba package seemed of interest, however annoyingly, they don't specify in the package how large is "large"!!!, and when you read the paper on which it is based, it deals with rather smaller data matrices than the ones I have, so it is unclear how it would perform and scale.
So, no one has direct experience running irlba on a data matrix as large as 500,000 x 1,000 or larger?
kind regards
Andy
________________________________
From: Berend Hasselman
Cc: "r-help@R-project.org"
Sent: Monday, 8 April 2013, 20:31
Subject: Re: [R] SVD on very large data matrix
>
>
> Dear All,
>
> I need to perform a SVD on a very large data matrix, of dimension ~ 500,000 x 1,000 , and I am looking
> for an efficient algorithm that can perform an approximate (partial) SVD to extract on the order of the top 50
> right and left singular vectors.
>
> Would be very grateful for any advice on what R-packages are available to perform such a task, what the RAM requirement is, and indeed what would be the state-of-the-art in terms of numerical algorithms and programming
> language to use to accomplish this task.
Info found with package sos and findFn("svd") and scrolling through the list for something relevant.
Have a look at package irlba.
It can work with dense matrices and sparse matrices as provided by package Matrix, according to the documentation.
Berend
[[alternative HTML version deleted]]