[Rd] cerr and cout not working calling c++ from R

Sean Robert McGuffee sean.mcguffee at gmail.com
Tue May 10 20:08:45 CEST 2011 

Just a short follow up to the list.
It turns out that the strange behavior I was experiencing had nothing to do
with cout and cerr, but ostreams in general. For some reason, at a minimum,
the operators: 
ostream& operator<< (long& val );
ostream& operator<< (double& val );
were not functioning properly and were causing my problem.
To get around this, I simply had to convert types like these to strings
before passing them into the ostreams with << operators.
I don't know if this was due to a mistake in which libraries I linked to or
which headers are being read when I make my project. However, it seems that
I have ruled out the possibility of this being unique to 64-bit longs as the
doubles are standard types on any machine. Anyway, I appreciate everyone's
help, and if anyone has trouble with this type of thing in the future, a
simple couple of functions I used to convert the long or double values to
strings were:
string getStringFromDouble(double d)
{
    char*doubleValue = (char*)calloc(1000,sizeof(char));
    sprintf(doubleValue,"%f",d);
    string returnValue(doubleValue);
    free(doubleValue);
    return(returnValue);
}
string getStringFromLong(long i)
{
    char*intValue = (char*)calloc(1000,sizeof(char));
    sprintf(intValue,"%ld",i);
    string returnValue(intValue);
    free(intValue);
    return(returnValue);
}
I'm not sure why this was necessary, but it solves my issue.
Thanks again for all the help--I learned a lot about streams and stream
buffers in the process.
Thanks,
Sean



On 5/8/11 12:56 AM, "Sean Robert McGuffee" <sean.mcguffee at gmail.com> wrote:

> Thanks, Simon,
> That link may prove to be very useful. I think what I can do is make a
> similar version of a buffer to one in that example and then have it
> conditionally write to Rprintf or cout and/or REprintf or cerr,
> depending on whether I'm running a c++ program from the command line
> or launching a function from R. Then I can just replace all of my cout
> and cerr terms with this new buffer or two in global space. Before I
> call those functions in R, I'll set them to write to R before I launch
> any code that uses it. This should allow me to leave all my code
> exactly how it is with a minor replacement of one or two terms for
> another. So, if the errors in my performance are only caused by
> writing to cout and/or cerr from R, then they can be completely
> avoided by all of those substitutions writing to Rprintf and REprintf.
> This seems like it is going to work exactly as I'm hoping.
> Thanks again,
> Sean
> 
> On Sat, May 7, 2011 at 6:13 PM, Simon Urbanek
> <simon.urbanek at r-project.org> wrote:
>> Sean,
>> 
>> just to clarify - this is not about something "not working" or any leaks.
>> cout/cerr do exactly what they are supposed to, it's just in most cases not
>> what you want. The main reason is that R does not necessarily use
>> stdin/stdout so cout/cerr may have nothing to do with the R console output.
>> The effects of using cout/cerr vary by the actual setup in which you run R as
>> I was explaining earlier. In the most complex setup (default for interactive
>> R on OS X, actually) you have:
>> 
>> 1) stdin/stdout/stderr - those are entirely independent of the R console
>> 2) ReadConsole/WriteConsole - this is how R interacts with the console
>> 3) C++ cin/cout/cerr - by default a buffered link to stdin/out/err
>> 
>> So as you see the reason for cout being strongly discouraged is simply
>> because it has no relationship with R and produces no output in R itself.
>> What makes things seemingly puzzling is that some UI implementations in R use
>> stdin/out/err in Read/WriteConsole, so suddenly you are dealing with three
>> independent streams and buffers (e.g., cout, WriteCondole and stdout all
>> eventually ending in the same place but at different times). If your final
>> output is a tty then they all get synchronized at the end into the same tty,
>> but since each of them has a buffer, the result is random at the best.
>> 
>> Due to the interactions, you must be careful not to mess things up with any
>> additional low-level stream manipulations. For example, the R.app GUI doesn't
>> use stdout/err for console at all, since it is driven by text views instead,
>> so it uses exclusively Read/WriteConsole (and so does the Windows GUI).
>> However, it also monitors stdout/err for external output (e.g. from system())
>> and uses two separate pipes to merge that output into the console with
>> different colors. So if you were to start to mess with the descriptors for
>> stdout/err you could break that setup.
>> 
>> That said, if you want a solution for C++ you should simply write a stream or
>> buffer that uses Rprintf/REprintf for output. I was assuming that others have
>> done that before, but Dirk's response suggests that it is not as common. If
>> you choose to do it, you may find this link helpful:
>> http://groups.google.com/group/comp.lang.c++/msg/1d941c0f26ea0d81
>> 
>> Cheers,
>> Simon
>> 
>> 
>> On May 7, 2011, at 12:42 AM, Sean Robert McGuffee wrote:
>> 
>>> This is a good idea as to converting to stringstream's because it
>>> could let me remove cout and cerr from the equation. At the moment, I
>>> get that feeling from the experts that having cout and cerr is causing
>>> a bug by not mixing with R. I have other cases where algorithms that
>>> avoid cout and cerr aren't working, but as far as my experience with
>>> bugs in c code goes, a memory issue in one place can make everything
>>> go haywire, and an random cout or cerr somewhere that I don't catch
>>> could maybe be the source of the issue. I also like this idea of
>>> piping and then calling Rprintf. Along the same lines, I could create
>>> a string and then either spit to cout or cerr or spit to Rprintf
>>> depending on some sort of setting. One of the key reasons I have been
>>> hanging on to cout and cerr is that I want to keep all of my c++ code
>>> identical to the status quo as far as the command line interface is
>>> concerned. On the other hand, maybe I can just globally substitute
>>> cout and cerr to random file streams and avoid the conflict with
>>> stdout and stderr. I take it that it isn't ostreams per say causing
>>> the problem but that it's something specific about cout and cerr not
>>> mixing with R. Then maybe I can pass their buffers to cout, cerr, or
>>> Rprintf, depending on preferences later. What I would really like to
>>> do maybe be beyond my grasp, but that would be to integrate ostreams
>>> into R. I am toying with an idea of mimicking cout and cerr with some
>>> new classes that are almost verbatim copies of cout and cerr with
>>> substituted names. Then maybe inside their internals I can more easily
>>> create conditional access to Rprintf calls. What I like about this
>>> strategy is that I could do a global substitution of cout and cerr
>>> with new objects of more useful functionality, and that way I might
>>> not have to re-write all my c++ code which constantly calls on these
>>> standard streams. Plus, if I get something generic to work, maybe I
>>> can pass it on to Dirk so others won't have to deal with this stuff in
>>> the future. I find this fascinating that the output may be tty. I see
>>> now that the device on my Mac that /dev/stdout points to is a tty. I
>>> have never encountered tty before, but I have a vague memory of a term
>>> like that back in the day of dial up! Oh the fun things I'm learning.
>>> Just to clarify though, is the only way my code "mixes IO systems
>>> between R and C++" by way of cout and cerr happening to be
>>> synchronized with stderr and stdout by default? Or is there something
>>> inherent in ostreams in general that doesn't mesh with R? I guess I
>>> find it a little bit surprising that calling
>>> ios_base::sync_with_stdio(false) and redirecting those streams to
>>> other files doesn't dodge that bullet. I know that doesn't break the
>>> ostreams themselves because it works fine from the command line
>>> interface. I guess I'll have to delve more into how c++ works behind
>>> the scenes to figure that part out. I think Dirk has a really good
>>> suggestion to comment out all of my cout and cerr calls and see if
>>> still get a bug. Obviously that wouldn't be any permanent fix for
>>> future users who want to link code with cout and cerr, but it would
>>> help me identify if this is the only source of error. It might take me
>>> till past the weekend to get to the bottom of some of these things,
>>> but I'll post what I find out.
>>> Thanks again--I appreciate everyone's useful suggestions.
>>> Sean
>>> 
>>> On Fri, May 6, 2011 at 10:44 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>>>> 
>>>> On 6 May 2011 at 22:09, Simon Urbanek wrote:
>>>> | Sean,
>>>> |
>>>> | the path form the console is not under your control and it depends
>>>> heavily on the internal settings in R (is console enabled, is R
>>>> interactive, is the output a tty or a file ...) - that's why you have to
>>>> use Rprint/REprintf - that's the only path that is guaranteed to work.
>>>> Anything else is settings-specific and not guaranteed to work (it may
>>>> appear to work in limited settings, but will not work in others). Note
>>>> that, for example, WriteConsole is only used if R is not fed via I/O pipes
>>>> - that may be what baffled you in that case.
>>>> |
>>>> | So AFAICS the only C++ solution (other than using Rprintf/REprintf
>>>> directly) is to create streams to pipes whose ends will call
>>>> Rprintf/REprintf with the content. Anything else is not general. You may
>>>> want to ask Rcpp people (Dirk is CC'd anyway ;)) - I'd imagine they must
>>>> have thought of it as it seems very natural to request in a C++ interface
>>>> ... (but I don't use Rcpp so I don't know).
>>>> 
>>>> Rcpp makes no claims about reimplementing IO, or providing new IO
>>>> facilities.
>>>> Rcpp is clearly bound by the R API upon which it builds; the R
>>>> documentation
>>>> (which I quoted in my earlier email in this thread) suggests to expect
>>>> nasty
>>>> things when one mixes IO systems between R and C++.
>>>> 
>>>> And that seems to be what Sean is experiencing; I for one am not surprised.
>>>> It was never clear from Seans's mails exactly how much time he has had for
>>>> reading of the available documentation, his approach seems to me mostly
>>>> driven by a vast and vague set of assumption on his part (eg the rout/rerr
>>>> business he comes back to), as well as a lot energy and a willingness to
>>>> just
>>>> try this.  Which is perfectly fine, but it can also fail.
>>>> 
>>>> I'd suggest to start small with as little IO as possible. Maybe even
>>>> comment
>>>> all IO out as much as possible. Or redo it as strstream and feed that to
>>>> Rprintf.
>>>> 
>>>> Dirk
>>>> 
>>>> 
>>>> | Cheers,
>>>> | Simon
>>>> |
>>>> |
>>>> |
>>>> | On May 6, 2011, at 3:19 PM, Sean Robert McGuffee wrote:
>>>> |
>>>> | > Hi,
>>>> | >
>>>> | > Thanks for the comments so far.
>>>> | >
>>>> | > I've been going through the code extensively and seem to be having
>>>> trouble
>>>> | > reproducing what R is doing, so this confuses me. For example, when I
>>>> write
>>>> | > out the pointer values for ptr_R_WriteConsole type functions, I can't
>>>> find a
>>>> | > function that matches what is actually being called.
>>>> | >
>>>> | > REprintf("Rstd_WriteConsole=%ld, ptr_R_WriteConsole=%ld,
>>>> | > R_WriteConsole=%ld\n", Rstd_WriteConsole, ptr_R_WriteConsole,
>>>> | > R_WriteConsole);
>>>> | >
>>>> | > It might be something like RTcl_WriteConsole too. However, I can't seem
>>>> to
>>>> | > find the headers I would need to try to find it's pointer value.
>>>> Anyway,
>>>> | > these are often hidden/protected/static which is an aspect of code that
>>>> I
>>>> | > only understand in concept. The concept I understand about this type of
>>>> | > stuff is that if you provide a better way of doing things, then maybe
>>>> you
>>>> | > hide some stuff that doesn't involve the better way. However, in this
>>>> case,
>>>> | > the hidden stuff is just making it difficult for me to debug and get to
>>>> | > bottom of what is going on. Anyway, I get different results if I call
>>>> on the
>>>> | > functions that simply use stdout when trying to reproduce Rprintf
>>>> | > functionality. I'm only doing that to try to get to the bottom of what
>>>> R is
>>>> | > doing with stdout and stderr. I also am having a similar issue with
>>>> trying
>>>> | > to understand what GNU GCC is doing with how it sync's cout and cerr
>>>> with
>>>> | > stdout and stderr. Maybe R is intercepting some of these components.
>>>> Anyway,
>>>> | >
>>>> | > I was hoping to get around the
>>>> | > "5.6 Interfacing C++ code
>>>> | >   ========================   [...]
>>>> | >      Using C++ iostreams, as in this example, is best avoided.  There
>>>> is
>>>> | >   no guarantee that the output will appear in the R console, and indeed
>>>> it
>>>> | >   will not on the R for Windows console.  Use R code or the C entry
>>>> points
>>>> | >   (*note Printing::) for all I/O if at all possible."
>>>> | > if at all possible. My reason is that I have 76386 lines of c++ code
>>>> that
>>>> | > I'm trying to interface with R, and it literally might be faster for me
>>>> to
>>>> | > update R rather than modify all of that code to suit R which seems to
>>>> be
>>>> | > lacking a c++ ostream interface.
>>>> | >
>>>> | > Ideally, I'd write rout and rerr versions of cout and cerr for R to
>>>> become
>>>> | > more compatible with c++. I have had some trouble figuring a lot of
>>>> this
>>>> | > hidden stuff out. Strategies I have considered are the following:
>>>> | > 1) Write macros that conditionally replace cerr and cout with Rprintf
>>>> | > somehow throughout all of my c++ code that I'm trying to interface with
>>>> R.
>>>> | > 2) Recreate the std::cerr and std::cout type of classes but maybe add
>>>> some
>>>> | > extra functionality for callbacks to when i/o events take place to
>>>> somehow
>>>> | > try to fix things.
>>>> | > 3) Make a scratch version of R source code that has no hidden aspects
>>>> to try
>>>> | > to investigate what is going on a little better.
>>>> | >
>>>> | > Also, I should mention that I found an unanswered post from someone who
>>>> had
>>>> | > a similar problem when trying to use cerr and cout when accessing
>>>> functions
>>>> | > from dynamically linked libraries. I wonder, is it possible that R
>>>> hasn't
>>>> | > completely connected to these libraries? I mean, it's doing some sort
>>>> of
>>>> | > magic where it finds a pointer to a function, but maybe c++ has a lot
>>>> going
>>>> | > on in the background related to these ostream types that could be
>>>> getting
>>>> | > messed up. Maybe some aspects of the global environment might be
>>>> different
>>>> | > somehow? I directly link to my libraries via a c++ compiler when I get
>>>> the
>>>> | > identical functions to work from main in c++.
>>>> | >
>>>> | > One final thing I'm considering is that the R manual also mentions
>>>> something
>>>> | > about garbage collection and some circumstances under which it's
>>>> important
>>>> | > to explicitly request pointers to be maintained. Is it possible that R
>>>> is
>>>> | > eating global pointers used by std::ostream ?
>>>> | >
>>>> | > All suggestions are valued and if anyone has recommendations on
>>>> strategy,
>>>> | > please let me know.
>>>> | >
>>>> | > Thanks,
>>>> | > Sean
>>>> | >
>>>> | >
>>>> | > On 5/6/11 2:50 PM, "Davor Cubranic" <cubranic at stat.ubc.ca> wrote:
>>>> | >
>>>> | >> On 2011-05-06, at 11:41 AM, Dirk Eddelbuettel wrote:
>>>> | >>
>>>> | >>> | I¹m trying to call some of my c++ code from R and seem to be having
>>>> an
>>>> | >>> issue
>>>> | >>> | with streams, although that¹s just one obvious sign of something
>>>> different
>>>> | >>> | in performance between calling the same function from main in c++
>>>> vs.
>>>> | >>> | calling the same function from R. [...]
>>>> | >>> In a nutshell, R 'owns' your console. That is part of the Faustian
>>>> bargain.
>>>> | >>
>>>> | >> But isn't he redirecting cout to a file, so the C++ output is not to
>>>> the
>>>> | >> console?
>>>> | >>
>>>> | >> Davor
>>>> | >
>>>> | >
>>>> | > On 5/6/11 2:41 PM, "Dirk Eddelbuettel" <edd at debian.org> wrote:
>>>> | >
>>>> | >>
>>>> | >> On 6 May 2011 at 14:21, Sean Robert McGuffee wrote:
>>>> | >> | Hi,
>>>> | >> |
>>>> | >> | Sorry, I just tried posting this but I had it in the wrong format of
>>>> text,
>>>> | >> | so this is a cleared format repost:
>>>> | >> |
>>>> | >> | I¹m trying to call some of my c++ code from R and seem to be having
>>>> an issue
>>>> | >> | with streams, although that¹s just one obvious sign of something
>>>> different
>>>> | >> | in performance between calling the same function from main in c++
>>>> vs.
>>>> | >> | calling the same function from R. I¹m not getting any signs of
>>>> errors from
>>>> | >> | R, but the behavior is different somehow and in a way that breaks my
>>>> | >> | algorithms. In both cases of launching a function from c++ or R, I
>>>> redirect
>>>> | >> | cout to a file using a streambuf. In particular, I can see progress
>>>> stop
>>>> | >> | when launching from R at a point when cout gets truncated when
>>>> starting to
>>>> | >> | write out iterator values in a std::vector<string>. Then, I find no
>>>> way to
>>>> | >> | write anything else to cout. When I write those values to Rprintf,
>>>> they are
>>>> | >> | there and they are as expected. I¹ve tried setting
>>>> | >> | ios_base::sync_with_stdio(false) in case R is messing with stdout
>>>> and stderr
>>>> | >> | somehow, but that made no difference. I¹m not quite sure how R uses
>>>> those C
>>>> | >> | FILE* streams. Anyway, I am wondering if anyone has a hunch as to
>>>> what might
>>>> | >> | be going on. At this point I don¹t know if it might be an issue with
>>>> R not
>>>> | >> | completely handling c++ from a library or if it is an issue with
>>>> cout in
>>>> | >> | particular or if there is something else I¹m missing. Please let me
>>>> know if
>>>> | >> | you have any suggestions for me.
>>>> | >>
>>>> | >> Doug Bates sometimes remarks that, once all other alternatives are
>>>> exhausted,
>>>> | >> one could consider reading the manual. And indeed, 'Writing R
>>>> Extensions' has
>>>> | >> this to say:
>>>> | >>
>>>> | >>   5.6 Interfacing C++ code
>>>> | >>   ========================
>>>> | >>
>>>> | >>   [...]
>>>> | >>
>>>> | >>      Using C++ iostreams, as in this example, is best avoided.  There
>>>> is
>>>> | >>   no guarantee that the output will appear in the R console, and
>>>> indeed it
>>>> | >>   will not on the R for Windows console.  Use R code or the C entry
>>>> points
>>>> | >>   (*note Printing::) for all I/O if at all possible.
>>>> | >>
>>>> | >> In a nutshell, R 'owns' your console. That is part of the Faustian
>>>> bargain.
>>>> | >>
>>>> | >> Dirk
>>>> | >>
>>>> | >> --
>>>> | >> Gauss once played himself in a zero-sum game and won $50.
>>>> | >>                      -- #11 at http://www.gaussfacts.com
>>>> | >
>>>> | > ______________________________________________
>>>> | > R-devel at r-project.org mailing list
>>>> | > https://stat.ethz.ch/mailman/listinfo/r-devel
>>>> | >
>>>> | >
>>>> |
>>>> 
>>>> --
>>>> Gauss once played himself in a zero-sum game and won $50.
>>>>                      -- #11 at http://www.gaussfacts.com
>>> 
>>> 
>> 
>>

More information about the R-devel mailing list