Rainer...I can't believe this did the trick.  You're a genius.  Thank you
sir.


On Thu, May 23, 2013 at 7:07 AM, Rainer Schuermann <
rainer.schuermann@gmx.net> wrote:

> Using the data generated with your code below, does
>
> rbind( DF1, DF2[ !(DF2$X.TIME %in% DF1$X.TIME), ] )
> DF1 <- DF1[ order( DF1$X.DATE, DF1$X.TIME ), ]
>
> do the job?
>
> Rgds,
> Rainer
>
>
>
>
> On Thursday 23 May 2013 05:54:26 Adeel - SafeGreenCapital wrote:
> > Thank you Blaser:
> >
> > This is the exact solution I came up with but when comparing 8M rows
> even on
> > an 8G machine, one runs out of memory.  To run this effectively, I have
> to
> > break the DF into smaller DFs, loop through them and then do a massive
> > rmerge at the end.  That's what takes 8+ hours to compute.
> >
> > Even the bigmemory package is causing OOM issues.
> >
> > -----Original Message-----
> > From: Blaser Nello [mailto:nblaser@ispm.unibe.ch]
> > Sent: Thursday, May 23, 2013 12:15 AM
> > To: Adeel Amin; r-help@r-project.org
> > Subject: RE: [R] adding rows without loops
> >
> > Merge should do the trick. How to best use it will depend on what you
> > want to do with the data after.
> > The following is an example of what you could do. This will perform
> > best, if the rows are missing at random and do not cluster.
> >
> > DF1 <- data.frame(X.DATE=rep(01052007, 7), X.TIME=c(2:5,7:9)*100,
> > VALUE=c(37, 42, 45, 45, 45, 42, 45), VALE2=c(29,24,28,27,35,32,32))
> > DF2 <- data.frame(X.DATE=rep(01052007, 7), X.TIME=c(2:8)*100,
> > VALUE=c(37, 42, 45, 45, 45, 42, 45), VALE2=c(29,24,28,27,35,32,32))
> >
> > DFm <- merge(DF1, DF2, by=c("X.DATE", "X.TIME"), all=TRUE)
> >
> > while(any(is.na(DFm))){
> >   if (any(is.na(DFm[1,]))) stop("Complete first row required!")
> >   ind <- which(is.na(DFm), arr.ind=TRUE)
> >   prind <- matrix(c(ind[,"row"]-1, ind[,"col"]), ncol=2)
> >   DFm[is.na(DFm)] <- DFm[prind]
> > }
> > DFm
> >
> > Best,
> > Nello
> >
> > -----Original Message-----
> > From: r-help-bounces@r-project.org [mailto:r-help-bounces@r-project.org]
> > On Behalf Of Adeel Amin
> > Sent: Donnerstag, 23. Mai 2013 07:01
> > To: r-help@r-project.org
> > Subject: [R] adding rows without loops
> >
> > I'm comparing a variety of datasets with over 4M rows.  I've solved this
> > problem 5 different ways using a for/while loop but the processing time
> > is murder (over 8 hours doing this row by row per data set).  As such
> > I'm trying to find whether this solution is possible without a loop or
> > one in which the processing time is much faster.
> >
> > Each dataset is a time series as such:
> >
> > DF1:
> >
> >     X.DATE X.TIME VALUE VALUE2
> > 1 01052007   0200    37     29
> > 2 01052007   0300    42     24
> > 3 01052007   0400    45     28
> > 4 01052007   0500    45     27
> > 5 01052007   0700    45     35
> > 6 01052007   0800    42     32
> > 7 01052007   0900    45     32
> > ...
> > ...
> > ...
> > n
> >
> > DF2
> >
> >     X.DATE X.TIME VALUE VALUE2
> > 1 01052007   0200    37     29
> > 2 01052007   0300    42     24
> > 3 01052007   0400    45     28
> > 4 01052007   0500    45     27
> > 5 01052007   0600    45     35
> > 6 01052007   0700    42     32
> > 7 01052007   0800    45     32
> >
> > ...
> > ...
> > n+4000
> >
> > In other words there are 4000 more rows in DF2 then DF1 thus the
> > datasets are of unequal length.
> >
> > I'm trying to ensure that all dataframes have the same number of X.DATE
> > and X.TIME entries.  Where they are missing, I'd like to insert a new
> > row.
> >
> > In the above example, when comparing DF2 to DF1, entry 01052007 0600
> > entry is missing in DF1.  The solution would add a row to DF1 at the
> > appropriate index.
> >
> > so new dataframe would be
> >
> >
> >     X.DATE X.TIME VALUE VALUE2
> > 1 01052007   0200    37     29
> > 2 01052007   0300    42     24
> > 3 01052007   0400    45     28
> > 4 01052007   0500    45     27
> > 5 01052007   0600    45     27
> > 6 01052007   0700    45     35
> > 7 01052007   0800    42     32
> > 8 01052007   0900    45     32
> >
> > Value and Value2 would be the same as row 4.
> >
> > Of course this is simple to accomplish using a row by row analysis but
> > with of 4M rows the processing time destroying and rebinding the
> > datasets is very time consuming and I believe highly un-R'ish.  What am
> > I missing?
> >
> > Thanks!
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
> > ______________________________________________
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]

