[R] Reshaping an array - how does it work in R

[Roy Mendelssohn - NOAA Federal]

> On Mar 19, 2016, at 8:18 AM, Henrik Bengtsson <henrik.bengtsson at gmail.com> wrote:
> 
> On Fri, Mar 18, 2016 at 8:28 PM, Roy Mendelssohn - NOAA Federal
> <roy.mendelssohn at noaa.gov> wrote:
>> Hi Henrik:
>> 
>> I want to do want in oceanography is called an EOF, which is just a PCA analysis. Unless I am missing something, in R I need to flatten my 3-D matrix into a 2-D data matrix. I can fit the entire 30GB matrix into memory, and I believe I have enough memory to do the PCA by constraining the number of components returned .  What I don’t think I have enough memory for is an operation that makes a copy of the matrix.
>> 
>> As I said, in theory I know how to do the flattening, it a simple command, but in practice I don’t have enough memory.  So I spent the afternoon rewriting my code to read in parts of the data at a time and then putting those in the appropriate places of a matrix already flattened of appropriate size.  In case someone is wondering, on the 3D grid the matrix is [1001,1001,3650].  So I create an empty matrix size [1001*1001,3650], and read in a slice of the lat-lon grid, and map those into the appropriate places in the flattened matrix.  By reading in appropriately sized chunks  my memory usage is not pushed too far.
> 
> Sounds good.  There's another small caveat. Make sure to specify the
> 'data' argument for matrix() we allocating an "empty" matrix, e.g.
> 
>    X <- matrix(NA_real_, nrow=1001*1001, ncol=3650)
> 
> This will give you a double matrix with all missing value.  If you use
> the default
> 
>    X <- matrix(nrow=1001*1001, ncol=3650)
> 
> you'll get a logical matrix, which will introduce a copy as soon as
> you assign a double value (e.g. X[1,1] <- 3.14). The latter is a
> complete waste of memory/time. See
> http://www.jottr.org/2014/06/matrixNA-wrong-way.html for details.
> 
> /Henrik

Thanks.  Yes one time for some reason I can’t remember I did ?NA  where that is documented but it is not something you would think of offhand.

-Roy


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.