[R] Big data and column correspondence problem
daniel at umd.edu
Wed Jul 27 08:45:15 CEST 2011
If A has more columns than in your example, you could always try to only
merge those columns of A with B that are relevant for the merging. You could
then cbind the result of the merging back together with the rest of A as
long as the merged data preserved the same order as in A.
Alternatively, you can always use chunks of A and do the merging separately,
e.g., for blocks of 10000 observations or so.
On a MacBook Pro with 4 Gs of RAM and a 2.4 GHz Duo Core processor it would
take you about 40 minutes if you do chunks for 15000 observations. I am not
sure whether the loop would be slower than that.
On a different note: how are you matching if AA has multiple matches in BB?
> Thanks Daniel, that helped me. Based on your suggestions I built this
> final code:
> AA = c(4,4,4,2,2,6,8,9)
> A1 = c(3,3,11,5,5,7,11,12)
> A2 = c(3,3,7,3,5,7,11,12)
> A = cbind(AA, A1, A2)
> BB = c(2,2,4,6,6)
> B1 =c(5,11,7,13,NA)
> B2 =c(4,12,11,NA,NA)
> B3 =c(12,13,NA,NA,NA)
> A = cbind(AA, A1, A2,0)
> newdata$dum <- rowSums (newdata[,matchcols(newdata,
> with=c("B"))]==newdata$A1, na.rm = FALSE, dims = 1)*
> rowSums (newdata[,matchcols(newdata, with=c("B"))]==newdata$A2, na.rm =
> FALSE, dims = 1)
> Which gives me the same result of the "loop" version. Unfortunately, I
> can't replicate it on the original data since i can't make the merge work:
> i get an error message "Reached total allocation of 4090Mb". So, I'm stuck
> If anyone could shed some light on this problem, i would really
View this message in context: http://r.789695.n4.nabble.com/Big-data-and-column-correspondence-problem-tp3694912p3697709.html
Sent from the R help mailing list archive at Nabble.com.
More information about the R-help