[Rd] cerr and cout not working calling c++ from R

[Sean Robert McGuffee]

Thanks, Simon,
That link may prove to be very useful. I think what I can do is make a
similar version of a buffer to one in that example and then have it
conditionally write to Rprintf or cout and/or REprintf or cerr,
depending on whether I'm running a c++ program from the command line
or launching a function from R. Then I can just replace all of my cout
and cerr terms with this new buffer or two in global space. Before I
call those functions in R, I'll set them to write to R before I launch
any code that uses it. This should allow me to leave all my code
exactly how it is with a minor replacement of one or two terms for
another. So, if the errors in my performance are only caused by
writing to cout and/or cerr from R, then they can be completely
avoided by all of those substitutions writing to Rprintf and REprintf.
This seems like it is going to work exactly as I'm hoping.
Thanks again,
Sean

On Sat, May 7, 2011 at 6:13 PM, Simon Urbanek
<simon.urbanek at r-project.org> wrote:
> Sean,
>
> just to clarify - this is not about something "not working" or any leaks. cout/cerr do exactly what they are supposed to, it's just in most cases not what you want. The main reason is that R does not necessarily use stdin/stdout so cout/cerr may have nothing to do with the R console output. The effects of using cout/cerr vary by the actual setup in which you run R as I was explaining earlier. In the most complex setup (default for interactive R on OS X, actually) you have:
>
> 1) stdin/stdout/stderr - those are entirely independent of the R console
> 2) ReadConsole/WriteConsole - this is how R interacts with the console
> 3) C++ cin/cout/cerr - by default a buffered link to stdin/out/err
>
> So as you see the reason for cout being strongly discouraged is simply because it has no relationship with R and produces no output in R itself. What makes things seemingly puzzling is that some UI implementations in R use stdin/out/err in Read/WriteConsole, so suddenly you are dealing with three independent streams and buffers (e.g., cout, WriteCondole and stdout all eventually ending in the same place but at different times). If your final output is a tty then they all get synchronized at the end into the same tty, but since each of them has a buffer, the result is random at the best.
>
> Due to the interactions, you must be careful not to mess things up with any additional low-level stream manipulations. For example, the R.app GUI doesn't use stdout/err for console at all, since it is driven by text views instead, so it uses exclusively Read/WriteConsole (and so does the Windows GUI). However, it also monitors stdout/err for external output (e.g. from system()) and uses two separate pipes to merge that output into the console with different colors. So if you were to start to mess with the descriptors for stdout/err you could break that setup.
>
> That said, if you want a solution for C++ you should simply write a stream or buffer that uses Rprintf/REprintf for output. I was assuming that others have done that before, but Dirk's response suggests that it is not as common. If you choose to do it, you may find this link helpful:
> http://groups.google.com/group/comp.lang.c++/msg/1d941c0f26ea0d81
>
> Cheers,
> Simon
>
>
> On May 7, 2011, at 12:42 AM, Sean Robert McGuffee wrote:
>
>> This is a good idea as to converting to stringstream's because it
>> could let me remove cout and cerr from the equation. At the moment, I
>> get that feeling from the experts that having cout and cerr is causing
>> a bug by not mixing with R. I have other cases where algorithms that
>> avoid cout and cerr aren't working, but as far as my experience with
>> bugs in c code goes, a memory issue in one place can make everything
>> go haywire, and an random cout or cerr somewhere that I don't catch
>> could maybe be the source of the issue. I also like this idea of
>> piping and then calling Rprintf. Along the same lines, I could create
>> a string and then either spit to cout or cerr or spit to Rprintf
>> depending on some sort of setting. One of the key reasons I have been
>> hanging on to cout and cerr is that I want to keep all of my c++ code
>> identical to the status quo as far as the command line interface is
>> concerned. On the other hand, maybe I can just globally substitute
>> cout and cerr to random file streams and avoid the conflict with
>> stdout and stderr. I take it that it isn't ostreams per say causing
>> the problem but that it's something specific about cout and cerr not
>> mixing with R. Then maybe I can pass their buffers to cout, cerr, or
>> Rprintf, depending on preferences later. What I would really like to
>> do maybe be beyond my grasp, but that would be to integrate ostreams
>> into R. I am toying with an idea of mimicking cout and cerr with some
>> new classes that are almost verbatim copies of cout and cerr with
>> substituted names. Then maybe inside their internals I can more easily
>> create conditional access to Rprintf calls. What I like about this
>> strategy is that I could do a global substitution of cout and cerr
>> with new objects of more useful functionality, and that way I might
>> not have to re-write all my c++ code which constantly calls on these
>> standard streams. Plus, if I get something generic to work, maybe I
>> can pass it on to Dirk so others won't have to deal with this stuff in
>> the future. I find this fascinating that the output may be tty. I see
>> now that the device on my Mac that /dev/stdout points to is a tty. I
>> have never encountered tty before, but I have a vague memory of a term
>> like that back in the day of dial up! Oh the fun things I'm learning.
>> Just to clarify though, is the only way my code "mixes IO systems
>> between R and C++" by way of cout and cerr happening to be
>> synchronized with stderr and stdout by default? Or is there something
>> inherent in ostreams in general that doesn't mesh with R? I guess I
>> find it a little bit surprising that calling
>> ios_base::sync_with_stdio(false) and redirecting those streams to
>> other files doesn't dodge that bullet. I know that doesn't break the
>> ostreams themselves because it works fine from the command line
>> interface. I guess I'll have to delve more into how c++ works behind
>> the scenes to figure that part out. I think Dirk has a really good
>> suggestion to comment out all of my cout and cerr calls and see if
>> still get a bug. Obviously that wouldn't be any permanent fix for
>> future users who want to link code with cout and cerr, but it would
>> help me identify if this is the only source of error. It might take me
>> till past the weekend to get to the bottom of some of these things,
>> but I'll post what I find out.
>> Thanks again--I appreciate everyone's useful suggestions.
>> Sean
>>
>> On Fri, May 6, 2011 at 10:44 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>>>
>>> On 6 May 2011 at 22:09, Simon Urbanek wrote:
>>> | Sean,
>>> |
>>> | the path form the console is not under your control and it depends heavily on the internal settings in R (is console enabled, is R interactive, is the output a tty or a file ...) - that's why you have to use Rprint/REprintf - that's the only path that is guaranteed to work. Anything else is settings-specific and not guaranteed to work (it may appear to work in limited settings, but will not work in others). Note that, for example, WriteConsole is only used if R is not fed via I/O pipes - that may be what baffled you in that case.
>>> |
>>> | So AFAICS the only C++ solution (other than using Rprintf/REprintf directly) is to create streams to pipes whose ends will call Rprintf/REprintf with the content. Anything else is not general. You may want to ask Rcpp people (Dirk is CC'd anyway ;)) - I'd imagine they must have thought of it as it seems very natural to request in a C++ interface ... (but I don't use Rcpp so I don't know).
>>>
>>> Rcpp makes no claims about reimplementing IO, or providing new IO facilities.
>>> Rcpp is clearly bound by the R API upon which it builds; the R documentation
>>> (which I quoted in my earlier email in this thread) suggests to expect nasty
>>> things when one mixes IO systems between R and C++.
>>>
>>> And that seems to be what Sean is experiencing; I for one am not surprised.
>>> It was never clear from Seans's mails exactly how much time he has had for
>>> reading of the available documentation, his approach seems to me mostly
>>> driven by a vast and vague set of assumption on his part (eg the rout/rerr
>>> business he comes back to), as well as a lot energy and a willingness to just
>>> try this.  Which is perfectly fine, but it can also fail.
>>>
>>> I'd suggest to start small with as little IO as possible. Maybe even comment
>>> all IO out as much as possible. Or redo it as strstream and feed that to
>>> Rprintf.
>>>
>>> Dirk
>>>
>>>
>>> | Cheers,
>>> | Simon
>>> |
>>> |
>>> |
>>> | On May 6, 2011, at 3:19 PM, Sean Robert McGuffee wrote:
>>> |
>>> | > Hi,
>>> | >
>>> | > Thanks for the comments so far.
>>> | >
>>> | > I've been going through the code extensively and seem to be having trouble
>>> | > reproducing what R is doing, so this confuses me. For example, when I write
>>> | > out the pointer values for ptr_R_WriteConsole type functions, I can't find a
>>> | > function that matches what is actually being called.
>>> | >
>>> | > REprintf("Rstd_WriteConsole=%ld, ptr_R_WriteConsole=%ld,
>>> | > R_WriteConsole=%ld\n", Rstd_WriteConsole, ptr_R_WriteConsole,
>>> | > R_WriteConsole);
>>> | >
>>> | > It might be something like RTcl_WriteConsole too. However, I can't seem to
>>> | > find the headers I would need to try to find it's pointer value. Anyway,
>>> | > these are often hidden/protected/static which is an aspect of code that I
>>> | > only understand in concept. The concept I understand about this type of
>>> | > stuff is that if you provide a better way of doing things, then maybe you
>>> | > hide some stuff that doesn't involve the better way. However, in this case,
>>> | > the hidden stuff is just making it difficult for me to debug and get to
>>> | > bottom of what is going on. Anyway, I get different results if I call on the
>>> | > functions that simply use stdout when trying to reproduce Rprintf
>>> | > functionality. I'm only doing that to try to get to the bottom of what R is
>>> | > doing with stdout and stderr. I also am having a similar issue with trying
>>> | > to understand what GNU GCC is doing with how it sync's cout and cerr with
>>> | > stdout and stderr. Maybe R is intercepting some of these components. Anyway,
>>> | >
>>> | > I was hoping to get around the
>>> | > "5.6 Interfacing C++ code
>>> | >   ========================   [...]
>>> | >      Using C++ iostreams, as in this example, is best avoided.  There is
>>> | >   no guarantee that the output will appear in the R console, and indeed it
>>> | >   will not on the R for Windows console.  Use R code or the C entry points
>>> | >   (*note Printing::) for all I/O if at all possible."
>>> | > if at all possible. My reason is that I have 76386 lines of c++ code that
>>> | > I'm trying to interface with R, and it literally might be faster for me to
>>> | > update R rather than modify all of that code to suit R which seems to be
>>> | > lacking a c++ ostream interface.
>>> | >
>>> | > Ideally, I'd write rout and rerr versions of cout and cerr for R to become
>>> | > more compatible with c++. I have had some trouble figuring a lot of this
>>> | > hidden stuff out. Strategies I have considered are the following:
>>> | > 1) Write macros that conditionally replace cerr and cout with Rprintf
>>> | > somehow throughout all of my c++ code that I'm trying to interface with R.
>>> | > 2) Recreate the std::cerr and std::cout type of classes but maybe add some
>>> | > extra functionality for callbacks to when i/o events take place to somehow
>>> | > try to fix things.
>>> | > 3) Make a scratch version of R source code that has no hidden aspects to try
>>> | > to investigate what is going on a little better.
>>> | >
>>> | > Also, I should mention that I found an unanswered post from someone who had
>>> | > a similar problem when trying to use cerr and cout when accessing functions
>>> | > from dynamically linked libraries. I wonder, is it possible that R hasn't
>>> | > completely connected to these libraries? I mean, it's doing some sort of
>>> | > magic where it finds a pointer to a function, but maybe c++ has a lot going
>>> | > on in the background related to these ostream types that could be getting
>>> | > messed up. Maybe some aspects of the global environment might be different
>>> | > somehow? I directly link to my libraries via a c++ compiler when I get the
>>> | > identical functions to work from main in c++.
>>> | >
>>> | > One final thing I'm considering is that the R manual also mentions something
>>> | > about garbage collection and some circumstances under which it's important
>>> | > to explicitly request pointers to be maintained. Is it possible that R is
>>> | > eating global pointers used by std::ostream ?
>>> | >
>>> | > All suggestions are valued and if anyone has recommendations on strategy,
>>> | > please let me know.
>>> | >
>>> | > Thanks,
>>> | > Sean
>>> | >
>>> | >
>>> | > On 5/6/11 2:50 PM, "Davor Cubranic" <cubranic at stat.ubc.ca> wrote:
>>> | >
>>> | >> On 2011-05-06, at 11:41 AM, Dirk Eddelbuettel wrote:
>>> | >>
>>> | >>> | I’m trying to call some of my c++ code from R and seem to be having an
>>> | >>> issue
>>> | >>> | with streams, although that’s just one obvious sign of something different
>>> | >>> | in performance between calling the same function from main in c++ vs.
>>> | >>> | calling the same function from R. [...]
>>> | >>> In a nutshell, R 'owns' your console. That is part of the Faustian bargain.
>>> | >>
>>> | >> But isn't he redirecting cout to a file, so the C++ output is not to the
>>> | >> console?
>>> | >>
>>> | >> Davor
>>> | >
>>> | >
>>> | > On 5/6/11 2:41 PM, "Dirk Eddelbuettel" <edd at debian.org> wrote:
>>> | >
>>> | >>
>>> | >> On 6 May 2011 at 14:21, Sean Robert McGuffee wrote:
>>> | >> | Hi,
>>> | >> |
>>> | >> | Sorry, I just tried posting this but I had it in the wrong format of text,
>>> | >> | so this is a cleared format repost:
>>> | >> |
>>> | >> | I’m trying to call some of my c++ code from R and seem to be having an issue
>>> | >> | with streams, although that’s just one obvious sign of something different
>>> | >> | in performance between calling the same function from main in c++ vs.
>>> | >> | calling the same function from R. I’m not getting any signs of errors from
>>> | >> | R, but the behavior is different somehow and in a way that breaks my
>>> | >> | algorithms. In both cases of launching a function from c++ or R, I redirect
>>> | >> | cout to a file using a streambuf. In particular, I can see progress stop
>>> | >> | when launching from R at a point when cout gets truncated when starting to
>>> | >> | write out iterator values in a std::vector<string>. Then, I find no way to
>>> | >> | write anything else to cout. When I write those values to Rprintf, they are
>>> | >> | there and they are as expected. I’ve tried setting
>>> | >> | ios_base::sync_with_stdio(false) in case R is messing with stdout and stderr
>>> | >> | somehow, but that made no difference. I’m not quite sure how R uses those C
>>> | >> | FILE* streams. Anyway, I am wondering if anyone has a hunch as to what might
>>> | >> | be going on. At this point I don’t know if it might be an issue with R not
>>> | >> | completely handling c++ from a library or if it is an issue with cout in
>>> | >> | particular or if there is something else I’m missing. Please let me know if
>>> | >> | you have any suggestions for me.
>>> | >>
>>> | >> Doug Bates sometimes remarks that, once all other alternatives are exhausted,
>>> | >> one could consider reading the manual. And indeed, 'Writing R Extensions' has
>>> | >> this to say:
>>> | >>
>>> | >>   5.6 Interfacing C++ code
>>> | >>   ========================
>>> | >>
>>> | >>   [...]
>>> | >>
>>> | >>      Using C++ iostreams, as in this example, is best avoided.  There is
>>> | >>   no guarantee that the output will appear in the R console, and indeed it
>>> | >>   will not on the R for Windows console.  Use R code or the C entry points
>>> | >>   (*note Printing::) for all I/O if at all possible.
>>> | >>
>>> | >> In a nutshell, R 'owns' your console. That is part of the Faustian bargain.
>>> | >>
>>> | >> Dirk
>>> | >>
>>> | >> --
>>> | >> Gauss once played himself in a zero-sum game and won $50.
>>> | >>                      -- #11 at http://www.gaussfacts.com
>>> | >
>>> | > ______________________________________________
>>> | > R-devel at r-project.org mailing list
>>> | > https://stat.ethz.ch/mailman/listinfo/r-devel
>>> | >
>>> | >
>>> |
>>>
>>> --
>>> Gauss once played himself in a zero-sum game and won $50.
>>>                      -- #11 at http://www.gaussfacts.com
>>
>>
>
>