I'm not an expert useR but I asked a similar question to stack overflow
that might give you new ideas.

http://stackoverflow.com/questions/17458556/how-can-i-speed-up-this-sapply-for-cross-checking-samples


On Thu, Aug 15, 2013 at 2:30 AM, Praveen Surendran
<ps629@medschl.cam.ac.uk>wrote:

> Dear Doran, Bert and Roger,
>
> Thank you for attending my query and for your valuable responses.
>
> The task is slightly more complex. Here's the real case... I have genetic
> variation data (40,000 single nucleotide polymorphisms) from 90,000
> individuals. This makes the 90,000 (samples) rows/columns of the matrix and
> 40,000 (SNPs) rows/columns of the matrix. Matrix data are genetic
> variations with values 0,1,2 or 3 where 0 is missing. There will be very
> few individuals with missing data.
>
> The task is to identify the relatedness between these 90,000 individuals
> using their genetic data (0,1,2 or 3). These values needs to be
> standardised before matrix multiplication. This will make the matrix much
> larger compared to the 0/1/2/3 matrix and most of these will be real
> numbers with decimals.
>
> Bert, I will not be doing a 90,000 x 40,000 %*% 40,000 x 90,000. The plan
> is to load this 90000 x 40000 matrix into R, then standardise and multiply
> this in batches of 90,000 samples against 500 samples using these 40,000
> variants and process these in parallel to get 90,000 x 90,000 comparisons.
> Does that sort of clarifies the situation?
>
> I tried loading a 90,000 x 40,000 matrix as a matrix in R this morning on
> the cluster with specifications described in my previous e-mail. This
> crashed due to memory overflow. I am trying for possibilities
>
> Any comments or thoughts will be greatly appreciated.
>
> Regards,
>
> Praveen.
>
> -----Original Message-----
> From: Roger Koenker [mailto:rkoenker@illinois.edu]
> Sent: 14 August 2013 23:06
> To: Praveen Surendran
> Cc: r-help@r-project.org
> Subject: Re: [R] Matrix Multiplication using R.
>
> In the event that these are moderately sparse matrices, you could try
> Matrix or SparseM.
>
>
> Roger Koenker
> rkoenker@illinois.edu
>
>
>
>
> On Aug 14, 2013, at 10:40 AM, Praveen Surendran wrote:
>
> > Dear all,
> >
> > I am exploring ways to perform multiplication of a 90000 x 40000 matrix
> with it's transpose.
> > As expected even a 40000 x 100 %*% 100x40000 didn't work on my
> desktop... giving the error "Error: cannot allocate vector of length
> 1600000000"
> >
> > However I am trying to run this on one node (64GB RAM; 2.60 GHz
> processor) of a high performance computing cluster.
> > Appreciate if anyone has any comments on whether it's advisable to
> perform a matrix multiplication of this size using R and also on any better
> ways to handle this task.
> >
> > Kind Regards,
> >
> > Praveen.
> >
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]

