Hi there,

I think the option of 30 seconds is ok because it is less than each one
expent reading the messages :-) Just kiding...

bests

milton

On Wed, Sep 2, 2009 at 8:01 PM, Leo Alekseyev <dnquark@gmail.com> wrote:

> Thanks everyone for the useful suggestions.  The bottleneck might be
> memory limitations of my machine (3.2GHz, 2 GB) and the fact that I am
> aggregating on a field that is a string.  Using the suggested
> as.data.frame(table(my.df$my.field)) I do get a speedup, but the
> computation still takes 30 seconds.  For the sake of comparison, I did
> write the "counting up rows with common values" function using a Perl
> hash (it's only 5 lines of Perl) and it takes 15 seconds to run -- a
> 2x speedup.  Not yet sure if it's worth the hassle.
>
> --Leo
>
> On Wed, Sep 2, 2009 at 4:28 PM, David M
> Smith<david@revolution-computing.com> wrote:
> > You may want to try using isplit (from the iterators package). Combined
> with
> > foreach, it's an efficient way of iterating through a data frame by
> groups
> > of rows defined by common values of a columns (which I think is what
> you're
> > after). You can speed things up further if you have a multiprocessor
> system
> > with the doMC package to run iterations in parallel. There's an example
> > here:
> >
> http://blog.revolution-computing.com/2009/08/blockprocessing-a-data-frame-with-isplit.html
> > Hope this helps,
> > # David Smith
> > On Wed, Sep 2, 2009 at 3:39 PM, Leo Alekseyev <dnquark@gmail.com> wrote:
> >>
> >> I have a data frame with about 10^6 rows; I want to group the data
> >> according to entries in one of the columns and do something with it.
> >> For instance, suppose I want to count up the number of elements in
> >> each group.  I tried something like aggregate(my.df$my.field,
> >> list(my.df$my.field), length) but it seems to be very slow.  Likewise,
> >> the split() function was slow (I killed it before it completed).  Is
> >> there a way to efficiently accomplish this in R?..  I am almost
> >> tempted to write an external Perl/Python script entering every row
> >> into a hashtable keyed by my.field and iterating over the keys...
> >> Might this be faster?..
> >>
> >> ______________________________________________
> >> R-help@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
> >> http://www.R-project.org/posting-guide.html<http://www.r-project.org/posting-guide.html>
> >> and provide commented, minimal, self-contained, reproducible code.
> >
> >
> >
> > --
> > David M Smith <david@revolution-computing.com>
> > Director of Community, REvolution Computing www.revolution-computing.com
> > Tel: +1 (206) 577-4778 x3203 (San Francisco, USA)
> >
> > Check out our upcoming events schedule at
> > www.revolution-computing.com/events
> >
> >
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html<http://www.r-project.org/posting-guide.html>
> and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]


