[R-SIG-Mac] multicore package: collecting results

[Ben Bolker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/29/2011 02:34 PM, Vincent Aubanel wrote:
> Hi all,
> 
> I'm using mclapply() of the multicore package for processing chunks
> of data in parallel --and it works great.
> 
> But when I want to collect all processed elements of the returned
> list into one big data frame it takes ages.
> 
> The elements are all data frames having identical column names, and
> I'm using a simple rbind() inside a loop to do that. But I guess it
> makes some expensive checking computations at each iteration as it
> gets slower and slower as it goes. Writing out to disk individual
> files, concatenating with the system and reading back from disk the
> resulting file is actually faster...

  Why do you have to write to disk?  Can you collect the results as a
list L and then do.call(rbind,L)  in one go?
> 
> Is there a magic argument to rbind() that I'm missing, or is there
> any other solution to collect the results of parallel processing
> efficiently?
> 
> Thanks, Vincent
> 
> _______________________________________________ R-SIG-Mac mailing
> list R-SIG-Mac at r-project.org 
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4Lc1sACgkQc5UpGjwzenMofgCdFEzN6qO8e8owV9GCJ6Mwafvn
n0oAoIZt5txoRS9Ma73XefnEzBDW19lE
=BDfd
-----END PGP SIGNATURE-----