[Rd] Small program embedding R crashes in 64 bits

[William]

Hi everyone,

I'm trying to create a small C++ program which embed R, but I'm having
problems when I try to do it on Windows 64 bits. I have created a
minimal reproducible example which is just the
src/gnuwin32/front-ends/rtest.c file with the R_ReplDLLdo1() loop, the
only difference is that I set the interactive mode to TRUE. Here is
the cpp file: https://gist.github.com/anonymous/08b42e83c949e250f60b068d58a3ec51

When compiled in 32 bits, everything works: I enter R commands and no
crash. When compiled in 64 bits (mingw64 and R x64 libs, and executed
with R x64 in the PATH), everything works except when there is an
error in R with a command entered by the user. Typically, entering "a"
shows "Error: object 'a' not found" and then the program immediately
crashes. Typing a stop() also trigger a crash.

Code returned by the program is 0xC0000028, which is STATUS_BAD_STACK
with the description: "An invalid or unaligned stack was encountered
during an unwind operation". I'm not really good at C++ or
makefile/compiler stuff, but I can't get it to work. I'm guessing this
as to do with some longjumps to return to the prompt when there is an
error but I don't know how to fix it.

Compiling in 32 bits:
P:/Rtools/mingw_32/bin/g++ -O3 -Wall -pedantic -IP:/R/R-3.4.3/include
-c testr.cpp -o testr.o
P:/Rtools/mingw_32/bin/g++ -o ./32.exe ./testr.o
-LP:/R/R-3.4.3/bin/i386 -lR -lRgraphapp

Results in:
C:\test> 32.exe
> a
Error: object 'a' not found
> # it works!

But compiling in 64 bits:
P:/Rtools/mingw_64/bin/g++ -O3 -Wall -pedantic -IP:/R/R-3.4.3/include
-c testr.cpp -o testr.o
P:/Rtools/mingw_64/bin/g++ -o ./64.exe ./testr.o
-LP:/R/R-3.4.3/bin/x64 -lR -lRgraphapp

Fails like this:
C:\test> 64.exe
> b <- 1
> b
[1] 1
> a
Error: object 'a' not found
<Program crashes with 0xC0000028 error, no prompt displayed>

I've tried lots of -std= flags, -DWIN64 -D_WIN64 and lots of other
defines I could find or think of but with no luck. What is missing?

Thanks,

William.