[R] Any better way of optimizing time for calculating distances in the mentioned scenario??

[Purna chander]

Dear All,

I'm dealing with a case, where 'manhattan' distance of each of 100
vectors is calculated from 10000 other vectors. For achieving this,
following 4 scenarios are tested:

1) scenario 1:
> x<-read.table("query.vec")
> v<-read.table("query.vec2")

> d<-matrix(nrow=nrow(v),ncol=nrow(x))
> for (i in 1:nrow(v)){
  + d[i,]<- sapply(1:nrow(x),function(z){dist(rbind(v[i,],x[z,]),method="manhattan")})
  + }
> print(d[1,1:10])

time taken for running the code is :
real    1m33.088s
user    1m32.287s
sys     0m0.036s

2) scenario2:

> x<-read.table("query.vec")
> v<-read.table("query.vec2")
> v<-as.matrix(v)
> d<-matrix(nrow=nrow(v),ncol=nrow(x))
> for (i in 1:nrow(v)){
   + tmp_m<-matrix(rep(v[i,],nrow(x)),nrow=nrow(x),byrow=T)
   + d[i,]<- rowSums(abs(tmp_m - x))
   + }
> print(d[1,1:10])

time taken for running the code is:
real    0m0.882s
user    0m0.854s
sys     0m0.025s

3) scenario3:

> x<-read.table("query.vec")
> v<-read.table("query.vec2")
> v<-as.matrix(v)
> d<-matrix(nrow=nrow(v),ncol=nrow(x))
> for (i in 1:nrow(v)){
  + d[i,]<-sapply(1:nrow(x),function(z){dist(rbind(v[i,],x[z,]),method="manhattan")})
  + }
> print(d[1,1:10])

time taken for running the code is:
real    1m3.817s
user    1m3.543s
sys     0m0.031s

4) scenario4:
> x<-read.table("query.vec")
> v<-read.table("query.vec2")
> v<-as.matrix(v)
> d<-dist(rbind(v,x),method="manhattan")
> m<-as.matrix(d)
> m2<-m[1:nrow(v),(nrow(v)+1):nrow(x)]
> print(m2[1,1:10])

time taken for running the code:
real    0m0.445s
user    0m0.401s
sys     0m0.041s


Queries:
1) Though scenario 4 is optimum, this scenario failed when matrix 'v'
having more no. of rows. An error occurred while converting distance
object 'd' to a matrix 'm'.
     For E.g: > m<-as.matrix(d)
       the above command resulted in error: "Error: cannot allocate
vector of size 922.7 MB".

So, what can be done to convert a larger dist object into a matrix or
how allocation size can be increased?

2) Here I observed that 'dist()' function calculates the distances
across all vectors present in a given matrix or dataframe. Is it not
possible to calculate distances of specific vectors from other vectors
present in a matrix using 'dist()' function? Which means, suppose if a
matrix 'x' having 20 rows, is it not possible using 'dist()' to
calculate only distance of 1st row vector from other 19 vectors.

3) Any other ideas to optimize the problem i'm facing with.

Regards,
Purnachander