Tim,

The thing is, I've got plenty of memory if each large matrix is stored on
each separate machine in the cluster just once. What I run into is that with
Rmpi type approaches the object gets copied up to 8 times (I have 8
effective cores per machine). What I am aiming for is speed, and starting to
use approaches that needlessly copy the object, then write these multiple
copies to disk does not seem efficient.

It seems to me what would be most efficient would be to fork the process
between machines using Rmpi and then within machine with multicore. How this
would all coordinate, however, is beyond me without manually splitting the
matrix up myself, which sort of defeats the idea of letting the packages do
the work for me.

Mark

Mark W. Kimpel MD  ** Neuroinformatics ** Dept. of Psychiatry
Indiana University School of Medicine

15032 Hunter Court, Westfield, IN  46074

(317) 490-5129 Work, & Mobile & VoiceMail
(317) 399-1219 Skype No Voicemail please


On Mon, Jan 4, 2010 at 2:39 PM, Tim Triche <tim.triche@gmail.com> wrote:

> almost forgot -- if you've got enough memory, the 'bigmemory' package may
> be easier to apply to this particular task than 'ff'.  both are worth
> exploring IMHO
>
>
> On Mon, Jan 4, 2010 at 11:38 AM, Tim Triche <tim.triche@gmail.com> wrote:
>
>> * multicore forks processes, which isn't always what you really want
>> * openMP is generally only useful in C extensions (see for example recent
>> 'beadarray' commits in the bioconductor tree); 'romp' is a bit decrepit
>> * ff might solve your problem a lot more easily!
>>
>> hope this helps
>>
>> --t
>>
>> On Mon, Jan 4, 2010 at 11:23 AM, Mark Kimpel <mwkimpel@gmail.com> wrote:
>>
>>> I recently ran into a problem that was easier to solve using mutilcore
>>> compared with Rmpi because I had a large matrix that I was performing
>>> calculations on and it copying it for each process ate up my 12GB of
>>> memory.
>>> I now have access to 2 Linux machine, each a core i7 with 12GB of memory,
>>> and wonder how I might speed things up even faster by using some sort of
>>> combination of multicore and Rmpi/snow.
>>>
>>> I'm a novice at this, so perhaps the answer is obvious, but is it
>>> possible
>>> to spawn to multiple machines with non-shared memory but within each
>>> machine
>>> used shared memory? If my novice understanding is correct, the former
>>> uses
>>> openMPI and the latter openMP.
>>>
>>> If this is possible, a self-contained example would be appreciated, even
>>> if
>>> the calculations are so trivial as to not make the parallelization
>>> worthwhile in the example case
>>>
>>> Mark
>>>
>>> Mark W. Kimpel MD  ** Neuroinformatics ** Dept. of Psychiatry
>>> Indiana University School of Medicine
>>>
>>> 15032 Hunter Court, Westfield, IN  46074
>>>
>>> (317) 490-5129 Work, & Mobile & VoiceMail
>>> (317) 399-1219 Skype No Voicemail please
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-hpc mailing list
>>> R-sig-hpc@r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>>>
>>
>>
>>
>> --
>> If a man will begin with certainties, he shall end in doubts;
>> but if he will be content to begin with doubts, he shall end in
>> certainties.
>>
>> Francis Bacon
>>
>
>
>
> --
> If a man will begin with certainties, he shall end in doubts;
> but if he will be content to begin with doubts, he shall end in
> certainties.
>
> Francis Bacon
>

	[[alternative HTML version deleted]]

