[Rd] R cmd check and multicore foreach loop

[Renaud Gaujoux]

Uhm... maybe this is the issue.
The issue seems to specially occur when I am building the vignette, 
which performs some parallel computations on a reduced example, 
therefore faster than in a normal usage of the package.
The two processes (on my dual core) output some logging information 
using cat, which are normally sent to the console, but I guess that in 
the case of a vignette these are written to tex file. It is very few 
data though (a loop counter), so writing should be also very quick, even 
in a file.

Could it be possible that a I/O deadlock happens because of this output?
I actually use mutexes, at the end of each loop to perform bigger 
writing operation on a common file, but I hadn't think these would be 
required for the logging output,  assuming that stdout and stderr were 
thread safe. Maybe I should use mutexes at this level too.
Does multicore or doMC provide optional separate logging as doMPI does? 
(I guess this might be better posted to R-hpc)

Thank you.
Renaud



On 17/08/2011 14:56, Brian G. Peterson wrote:
> On Wed, 2011-08-17 at 04:50 -0700, Tim Triche, Jr. wrote:
>> I'll see if I can put together a self-contained example.  Primarily,
>> the times that I use multicore (and attempted to use doSMP, mostly
>> because one of my users refuses to ditch Windows) are when I am
>> reading a ton of binary files, none of which depend on the others.
>> This is a blindingly obvious use-case for e.g. doMC and doSMP, yet
>> what typically happens is that the entire operation wedges.  I'm told
>> that doSMP really only works well with Revolution R, but per above, I
>> will try to put together a working self-contained example to show
>> how.
> Remember that physical I/O can bind up the processes too.  Having a
> bunch of processes all trying to read from local disk at the same time
> (especially when they are all trying to read the same file(s), a problem
> it seems you may not have) is a recipe for I/O locks that can seize up
> your processes.
>
> So, if the problem only occurs with physical I/O, the first thing I
> would try is to move that storage to a storage device on another machine
> that is tuned for that level of disk I/O.
>
> Regards,
>
>     - Brian
>

 

###
UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mai...{{dropped:5}}