I am doing an analysis that requires me to calculate correlations for a matrix of 15,000 rows x 50 columns.  For each row I want to calculate the correlation to all other rows and then for each row, find the n (say 10) most correlated rows.  If read in the 15,000 x 50 data from file and pass it to 'cor', this function quite appropriately (and very quickly) calculates all possible row by row comparisons and outputs a matrix of the results.  The problem is that this matrix is exceedingly large (approx 1GB).  I want to run this analysis thousands of times on a cluster and thus each job must be below 1GB (otherwise I'd just do it on a large memory machine - where it works fine).  Since I am only interested in the top n correlations for each row, I would prefer to only store these correlations.  However, If I use a loop strategy to calculate correlations and only keep the ones I want, it runs extremely slowly.  The correlations for one row (versus all others) actually takes as long as all rows versus all rows using the non-loop strategy!  Two questions: 

1) Does this performance difference make sense?  I expected looping to be slower but not that much slower.
2) Is there a way that I can pass the data matrix to 'cor' but only get back the top n correlations for each row in the output matrix?  Or, is there another way to get correlations quickly but only store the best results? 

Any help would be greatly appreciated.  Obi

#The nice R way to get all possible correlations quickly - too much memory used
file1 = read.table("test.txt", header=F, quote="", sep="\t", comment.char="", as.is=1)
file1_cor = cor(t(file1), method = "pearson", use = "pairwise.complete.obs")
diag(file1_cor) = NA #Set correlation to self as NA
for (i in 1:15000){
  corrs=file1_cor[,i]
  corrs_ordered=order(corrs,decreasing=TRUE)  #Order correlations from largest to smallest
  top_corrs=corrs[corrs_ordered[1:n]] #Get top n correlations - these would be added to some data structure and used for subsequent analysis
}

#The not so nice way to get all possible correlations for each row and then store only those that I want to keep. - too slow
file1 = read.table("test.txt", header=F, quote="", sep="\t", comment.char="", as.is=1)
for (i in 1:15000){
  corrs = vector(length=15000)
  for (j in 1:15000){
  cor_ij = cor(as.numeric(file1[i,]), as.numeric(file1[j,]), method = "pearson", use = "pairwise.complete.obs")
  corrs[j]=file1_gene_cor
  }
  corrs[i]=NA; #Set correlation to self as NA
  corrs_ordered=order(corrs,decreasing=TRUE)  #Order correlations from largest to smallest
  top_corrs=corrs[corrs_ordered[1:n]] #Get top n correlations - these would be added to some data structure and used for subsequent analysis
}


	[[alternative HTML version deleted]]


