[R] Working with large matrix

[A Ezhil]

HI Steve,

Thanks a lot for your suggestions. I never used bigmemory and biglm before. Is there a way (using systems commands) to combine awk and R? I was thinking of calling a R program inside Awk, passing each row to R, do the lm, get back results and process it with awk. Will this make the calculation fast? If you have any experience with large matrices, could you please share it with me?

Thanks again,
Ezhil

--- On Fri, 9/11/09, Steve Lianoglou <mailinglist.honeypot at gmail.com> wrote:

> From: Steve Lianoglou <mailinglist.honeypot at gmail.com>
> Subject: Re: [R] Working with large matrix
> To: "A Ezhil" <ezhil02 at yahoo.com>
> Cc: R-help at r-project.org
> Date: Friday, September 11, 2009, 11:56 PM
> Hi,
> 
> On Sep 11, 2009, at 1:15 PM, A Ezhil wrote:
> 
> > Dear All,
> > 
> > I have large matrix (46000 x 11250). I would like to
> do the linear regression for each row. I wrote a simple
> function that has lm() and used apply(mat,1,func). The issue
> is that it takes ages to load the file and also to finish
> the lm. I am using LINUX 64 bit with 32G mem. Is there an
> elegant and fast way of completing this task?
> 
> Do bigmemory and biglm help?
> 
> http://cran.r-project.org/web/packages/bigmemory
> http://cran.r-project.org/web/packages/biglm
> 
> -steve
> 
> --
> Steve Lianoglou
> Graduate Student: Computational Systems Biology
>   |  Memorial Sloan-Kettering Cancer Center
>   |  Weill Medical College of Cornell University
> Contact Info: http://cbio.mskcc.org/~lianos/contact
> 
>