[Rd] Thread synchronization [Was: Interrupting C++ code execution]

[Simon Urbanek]

Sean,

On Apr 27, 2011, at 3:21 PM, Sean Robert McGuffee wrote:

> Hi Simon,
> That makes a lot of sense to me. I'll start reading about R's event loop signaling. I'm not sure what the best method will be for me to flag the completeness of a threaded process in my case. In abstract it seems that I could get R's event loop to look for any type of flag. I think key for me in this case will be identifying whether a particular file has been completely produced or not. In principle I could put that type of info into the file itself, but I think I could also make a temp file somewhere with it's full path and flag info about it. Then the event loop could look for a particular pattern of temp file names. On the other hand, if I pass in that info when I start the event loop, that might work too.

Usually, the easiest on unix is to register a file handle as input handler (addInputHandler) - in practice a pipe - one end is owned by the thread and the other is owned by R. Then all you need is to write anything on the thread's end and it will wake up R's even loop and let you handle the read on that end so you can do anything. You could even have multiple threads share this one pipe since you could distinguish by payload which thread is calling. One example of this is the integrated HTTP server in R - see Rhttpd sources (it has also a variant that works on Windows using synchronization via OS event loop).


> Regarding the external pointer idea, I was thinking about passing an object to R as a return value after launching the thread, and then I might be able to access a pointer inside that object to reference it from my thread. That could be a binary vector or any type of object if I can figure out how to get to it from my thread. Honestly, I don't know much about dynamic referencing of objects from separate threads, but in principle memory is shared in this case. I'll let you know if I come up with anything generic... Please keep me posted on your  package. Are any versions of it available yet?

Yes, it is not released yet since it's not quite complete, but here we go, at your own risk ;):

http://rforge.net/threads

It will work on all platforms, eventually, but currently only unix is supported. The idea is sort of taking the multicore paradigm (parallel + collect) but using threads (threadEval + yield). The documentation it currently non-existent, but I plan to write a vignette for it ... maybe later this week ...

Cheers,
Simon


> It didn't happen to come up on my list of R packages. I haven't necessarily been maintaining an up-to-date version of R though. I don't know if that influences the package list it shows me.
> Sean
> 
> 
> On 4/26/11 8:51 PM, "Simon Urbanek" <simon.urbanek at r-project.org> wrote:
> 
>> Sean,
>> 
>> On Apr 26, 2011, at 5:06 PM, Sean Robert McGuffee wrote:
>> 
>>> I've been thinking about how to handle c++ threads that were started via Rcpp
>>> calls to some of my c++ libraries from R. My main obstacle is trying to make
>>> sure that users don't try to process files that are being generated by a
>>> thread before the thread finishes. One thing I am considering is having my
>>> threaded code return a class to R that contains a pointer that it remembers.
>>> Then maybe I could just change the value at that pointer when my thread
>>> finishes. Does that seem like a reasonable approach? I'm not completely sure
>>> if this is related to your issue or not, but it might be similar enough to be
>>> worth asking...
>> 
>> It depends. For a simple flag it's actually much more simple than that - you
>> can create a boolean vector (make sure you preserve it) and just update its
>> value when it's done - you don't even need an external pointer for that (if
>> your'e careful).
>> 
>> But the slight problem with that approach is rather that you don't have a way
>> to tell R about the status change, so essentially you can only poll on the R
>> side. A more proper way to deal with this is to use the event loop signaling
>> to signal in R that the flag has changed. I'm working on a "threads" package
>> that should help with that, but it's not complete yet (you can spawn threads
>> from R and you can actually even synchronize them with R [so if the result is
>> all you want it's there], but semaphores are not implemented yet  --- your
>> inquiry should shift it further up on my todo stack ;)).
>> 
>> Cheers,
>> Simon
> 
> 
>