[Rd] Single-threaded aspect

[Simon Urbanek]

As others said XPtr is not something in R so Rcpp mailing list would be the right place for that aspect.

However, it you forget Rcpp and phrase it as an R question, you also get much closer to the reason and answer. SEXP type is the internal representation of all objects in R. I assume your question is which operations in the R API on those are thread-safe. The answer is that most of them are not, the main reason being that the memory management is not thread-safe, i.e. you cannot allocate anything without synchronization. Since almost all API calls involve some memory allocations, they are not thread-safe. You can, however, allocate objects and the operate on their payload, e.g., you can get numerical input vectors, allocate the result vector and then perform your threaded computation in C on those, synchronize and get back - that's how most implicit parallel operations in R work (leveraging BLAS, OpenMP, etc.). That is also what Dirk replied in your SO answer (quote: "Packages like RcppParallel are very careful about using non-R data structures for multithreaded work."). Note that the payload of most native vectors (integer, real, complex) is technically non-R data structure in the sense so you can operate on those directly (some read-only operations are also thread-safe in the API as long as they can't trigger errors/warning/side-effects).

For completeness, memory allocation is not the only reason or obstacle for thread-safe R API calls, but a main one. Other issues involve error handling (you may long-jump out of your thread stack) and global state (devices, connections etc.). In short, it's not something that can be really solved without complete re-design and re-write.

Cheers,
Simon


> On May 12, 2016, at 9:16 AM, Charles Determan <cdetermanjr at gmail.com> wrote:
> 
> Thanks for the replies.  Regarding the answer by Dirk, I didn't feel like I
> still understood the reasoning why mclapply or foreach cannot handle XPtr
> objects.  Instead of cluttering the SO question with comments I was getting
> the impression that this was a limitation inherited with R objects (which
> XPtr is supposed to be a proxy for an R object according to Dirk's
> comment).  If this is not the case, I could repost this on Rcpp-devel
> unless it could be migrated.
> 
> Regards,
> Charles
> 
> On Thu, May 12, 2016 at 8:11 AM, Mark van der Loo <mark.vanderloo at gmail.com>
> wrote:
> 
>> Charles,
>> 
>> 1. Perhaps this question is better directed at the R-help or
>> R-pacakge-devel mailinglist.
>> 
>> 2. It basically means that R itself can only evaluate one R expression at
>> the time.
>> 
>> The parallel package circumvents this by starting multiple R-sessions and
>> dividing workload.
>> 
>> Compiled code called by R (such as C++ code through RCpp or C-code through
>> base R's interface) can execute multi-threaded code for internal purposes,
>> using e.g. openMP. A limitation is that compiled code cannot call R's C API
>> from multiple threads (in many cases). For example, it is not thread-safe
>> to create R-variables from multiple threads running in C. (R's variable
>> administration is such that the order of (un)making them from compiled code
>> matters).
>> 
>> I am not very savvy on Rcpp or XPtr objects, but it appears that Dirk
>> provided answers about that in your SO-question.
>> 
>> Best,
>> Mark
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Op do 12 mei 2016 om 14:46 schreef Charles Determan <cdetermanjr at gmail.com
>>> :
>> 
>>> R Developers,
>>> 
>>> Could someone help explain what it means that R is single threaded?  I am
>>> trying to understand what is actually going on inside R when users want to
>>> parallelize code.  For example, using mclapply or foreach (with some
>>> backend) somehow allows users to benefit from multiple CPUs.
>>> 
>>> Similarly there is the RcppParallel package for RMatrix/RVector objects.
>>> But none of these address the general XPtr objects in Rcpp.  Some readers
>>> here may recognize my question on SO (
>>> 
>>> http://stackoverflow.com/questions/37167479/rcpp-parallelize-functions-that-return-xptr
>>> )
>>> where I was curious about parallel calls to C++/Rcpp functions that return
>>> XPtr objects.  I am being a little more persistent here as this limitation
>>> provides a very hard stop on the development on one of my packages that
>>> heavily uses XPtr objects.  It's not meant to be a criticism or intended
>>> to
>>> be rude, I just want to fully understand.
>>> 
>>> I am willing to accept that it may be impossible currently but I want to
>>> at
>>> least understand why it is impossible so I can explain to future users why
>>> parallel functionality is not available.  Which just echos my original
>>> question, what does it mean that R is single threaded?
>>> 
>>> Kind Regards,
>>> Charles
>>> 
>>>        [[alternative HTML version deleted]]
>>> 
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>> 
>> 
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>