[Rd] [External] svd For Large Matrix

[Prof Brian Ripley]

On 13/08/2021 15:58, luke-tierney using uiowa.edu wrote:
> [copying the list]
> 
> svd() does support matrices with long vector data. Your example works
> fine for me on a machine with enough memory with either the reference
> BLAS/LAPACK or the BLAS/LAPACK used on Fedora 33 (flexiblas backed, I
> believe, by a version of openBLAS). Take a look at sessionInfo() to
> see what you are using and consider switching to another BLAS/LAPACK
> if necessary. Running under gdb may help tracking down where the issue
> is and reporting it for the BLAS/LAPACK you are using.

See also 
https://cran.r-project.org/doc/manuals/r-devel/R-ints.html#Large-matrices which 
(to nuance Prof Tierney's comment) mentions that svd on long-vector 
*complex* data has been known to segfault (with the reference BLAS/Lapack).

My guess was that this was an out-of-memory condition not handled 
elegantly by the OS.  (There are many reasons why the posting guide asks 
for the output of sessionInfo().)

We do not have the statistical context but it seems unlikely that anyone 
is interested in each of the 45m samples, and for information on the 
proteins a quite small sample of cells would suffice.  And that not all 
45m left singular values are required (most likely none are, in which 
case the underlying Lapack routine can use a more efficient calculation).

> 
> Best,
> 
> luke
> 
> On Fri, 13 Aug 2021, Dario Strbenac via R-devel wrote:
> 
>> Good day,
>>
>> I have a real scenario involving 45 million biological cells (samples) 
>> and 60 proteins (variables) which leads to a segmentation fault for 
>> svd. I thought this might be a good example of why it might benefit 
>> from a long vector upgrade.
>>
>> test <- matrix(rnorm(45000000*60), ncol = 60)
>> testSVD <- svd(test)
>>
>> *** caught segfault ***
>> address 0x7fe93514d618, cause 'memory not mapped'
>>
>> Traceback:
>> 1: La.svd(x, nu, nv)
>> 2: svd(test)



-- 
Brian D. Ripley,                  ripley using stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford