Thanks all for your helpful suggestions. I replaced the hard drive of this
computer and augmented the RAM to 8 GB over the weekend, and things seen to
be running very smoothly now. I'll certainly use the methods provided here
as I push the limits on these types of matrices and objects.
Best
*Ben Caldwell*
On Sun, Apr 21, 2013 at 9:21 AM, Charles Berry wrote:
> Benjamin Caldwell berkeley.edu> writes:
>
> >
> > Dear R helpers
> >
> > Reproducible example:
> >
> > #warning - this causes a hard freeze on the machines I've tried it on
> > matrix.holder<- matrix(rnorm(150), nrow=30, ncol=5)
> >
> > Out=
> > expand.grid(matrix.holder[,1],matrix.holder[,2],matrix.holder[,3],
> matrix.holder[,4],
> > matrix.holder[,5])
> >
> > Problem:
> >
> > I'm running an analysis that I would like to do using a matrix containing
> > all the possible combinations of the elements in a [30,5] matrix.
> Briefly,
> > each possible combination is used to index and subset another matrix. I
> > then run some models on the data in the subsetted matrix and then
> > sometimes
> > export the model results based on a couple criteria. 24,300,000
> > combinations seems to be too big for R on my computer (Intel i5, about
> 2.5
> > GB RAM free, 4 GB total, Rx64 2.15 ) to handle.
> >
> > Requests:
> >
> [snip]
>
>
>
> > I'd like to attempt to multithread [snip]
>
>
> Ben,
>
> The problem you have is "embarassingly parallel" - as they say.
>
> You can effectively use brute force solutions to parallelize the job
> and do it with subjobs that have smaller memory requirements.
>
> One way to parallelize the problem is to create the object 'matrix.holder',
> then loop thru the values of matrix.holder[,1] and create a subjob that
> will run all the computations for matrix.holder[i,1] and all the
> combinations of matrix.holder[,-1]. Run the subjob in a new process and
> save
> the results. Later on you combine the saved results.
>
> Also, you could try to run each subjob using parallel::mclapply() or
> some other parallelizing package. Or you could loop over each of the
> first two columns of matrix.holder creating 900 subjobs. Also, this gives
> you still smaller memory requirements for the individual jobs.
>
> HTH,
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]