[Rd] Embedded R, last errormessage, and stack smashing

[Laurent Gautier]

Dear list,

I am having an embedded R, dying with
*** stack smashing detected *** in one specific case.

My code is such as I evaluate R expression with C code like

res = R_tryEval(expr, env, &error);

and in case of error, get the error message (usually sucessfully) with
code like below:

SEXP geterrmessage = findVar(install("geterrmessage"), R_BaseNamespace);
PROTECT(expr = allocVector(LANGSXP, 1));
SETCAR(expr, geterrmessage);
PROTECT(res = Rf_eval(expr, R_GlobalEnv));
// ---> *** stack smashing detected ***

The call to Rf_eval does not return as the stack smashing error stops the show.

Tracing with gdb where the problem occurs, I follow the path:
eval -> R_CheckStack in main/error.c -> errorcall in main/error.c


In errorcall, I narrow down the origin of the problem around the lines:
658    va_start(ap, format);
659    verrorcall_dflt(call, format, ap);
660    va_end(ap);
and add a breakpoint on each one of those lines.


My gdb session till the stack smashing crashing is then looking like:

Breakpoint 7, Rf_errorcall (call=0x828ef8, format=0x7f4dbdc87698 "C
stack usage is too close to the limit")
    at errors.c:658
658	    va_start(ap, format);
(gdb) p ap
$2 = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x41c22f60,
reg_save_area = 0x41c22e70}}
(gdb) p format
$3 = 0x7f4dbdc87698 "C stack usage is too close to the limit"
(gdb) continue
Continuing.

Breakpoint 8, Rf_errorcall (call=0x828ef8, format=0x7f4dbdc87698 "C
stack usage is too close to the limit")
    at errors.c:659
659	    verrorcall_dflt(call, format, ap);
(gdb) continue
Continuing.

Breakpoint 6, Rf_errorcall (call=0x828ef8, format=0x7f4dbdc87406 "%s")
at errors.c:648
648	    if (R_ErrorHook != NULL) {
(gdb) continue
Continuing.

Breakpoint 7, Rf_errorcall (call=0x828ef8, format=0x7f4dbdc87406 "%s")
at errors.c:658
658	    va_start(ap, format);
(gdb) continue
Continuing.

Breakpoint 8, Rf_errorcall (call=0x828ef8, format=0x7f4dbdc87406 "%s")
at errors.c:659
659	    verrorcall_dflt(call, format, ap);
(gdb) continue
Continuing.
*** stack smashing detected ***: /usr/bin/python terminated


My understanding is that there is a C stack problem about to happen R
tries to report, but in the process of reporting it
causes a C stack-related crash. Is there something odd with R handling
the situation, or should I look for the cause
of the problem elsewhere ?

Thanks,


L.