[Rd] R segfault in fgets from do_system under high memory use (PR#14008)

[atp at piskorski.com]

Full_Name: Andrew Piskorski
Version: R 2.9.2 (Patched), 2009-09-24, svn.rev 49930, x86_64-unknown-linux-gnu
OS: Linux, Ubuntu 8.04.3 LTS
Submission from: (NULL) (66.31.65.247)


I have a large memory test case which segaults R everytime in an fgets
call from R's do_system (see below).

This appears to be because R does not check the return value of the
system popen, and I have a simple patch to src/main/sysutils.c and
src/unix/sys-unix.c which fixes the problem.  I will attempt to attach
the patch after submitting this initial bug report.

This is on Linux, Ubuntu 8.04.3 LTS with:

R 2.9.2 (Patched), 2009-09-24, svn.rev 49930, x86_64-unknown-linux-gnu


Below is some further detail on the problem, from BEFORE applying my
patch:
------------------------------------------------------------

Valgrind doesn't seem to find anything unusual until an "Invalid read
of size 4" in fgets right before it segfaults.  Valgrind is also
reporting an "Address 0x0 is not stack'd" message there, which I think
means that do_system is passing a 0 address to fgets, which is then
causing the segfault.

Looking at the fgets all in src/unix/sys-unix.c, the buf argument is
statically allocated so I don't see how it could be 0.  fp, the 3rd
argument to fgets, is set by R_popen().  So, I think the system
popen() call is failing to fork or allocate memory or whatever it's
trying to do, and is returning a NULL.

gdb and Valgrind output from the failure follow:
------------------------------------------------------------

Program received signal SIGSEGV, Segmentation fault.
(gdb) bt
#0  0x00007f735647f4fd in fgets () from /lib/libc.so.6
#1  0x00007f7356b39fe3 in do_system (call=<value optimized out>,
    op=<value optimized out>, args=<value optimized out>,
    rho=<value optimized out>) at ../../../src/unix/sys-unix.c:273
#2  0x00007f7356aa1c09 in do_internal (call=<value optimized out>,
    op=<value optimized out>, args=0xace1d220, env=0xa90e3820)
    at ../../../src/main/names.c:1150
#3  0x00007f7356a6ec11 in Rf_eval (e=0x872638, rho=0xa90e3820)
    at ../../../src/main/eval.c:461
[...]
#90 0x00007f7356a92950 in run_Rmainloop () at ../../../src/main/main.c:966
#91 0x000000000040088b in main (ac=<value optimized out>,
    av=<value optimized out>) at ../../../src/main/Rmain.c:33
#92 0x00007f735643a1c4 in __libc_start_main () from /lib/libc.so.6
#93 0x00000000004007a9 in _start ()
(gdb) q


==27499== Invalid read of size 4
==27499==    at 0x55E84FD: fgets (in /lib/libc-2.7.so)
==27499==    by 0x4FB6FB2: do_system (sys-unix.c:273)
==27499==    by 0x4F1EBD8: do_internal (names.c:1150)
==27499==    by 0x4EEBBE0: Rf_eval (eval.c:461)
==27499==    by 0x4EEC9D1: do_begin (eval.c:1191)
==27499==    by 0x4EEBBE0: Rf_eval (eval.c:461)
==27499==    by 0x4EEE34E: Rf_applyClosure (eval.c:667)
==27499==    by 0x4EEBAFB: Rf_eval (eval.c:505)
==27499==    by 0x4EEC9D1: do_begin (eval.c:1191)
==27499==    by 0x4EEBBE0: Rf_eval (eval.c:461)
==27499==    by 0x4EEBBE0: Rf_eval (eval.c:461)
==27499==    by 0x4EEC9D1: do_begin (eval.c:1191)
==27499==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
/home/andy/t/vg-R.sh: line 34: 27499 Segmentation fault