[R-pkg-devel] Issue with flang-new (segfault from C stack overflow)

[Ivan Krylov]

В Mon, 18 Dec 2023 11:06:16 +0100
Jisca Huisman <jisca.huisman using gmail.com> пишет:

> I isolated the problem in a minimal working example available here: 
> https://github.com/JiscaH/flang_segfault_min_example . All that does
> is pass a vector of length N*N back and forth between R and Fortran.
> It works fine for very long vectors (tested up to length 5e8), but
> throws a segfault when I reshape a large array in Fortran to a vector
> to pass to R, both when using RESHAPE() and when using loops.

You've done an impressive amount of investigative work. Thank you for
reducing your problem to such a small example! My eyes are drawn to
these two lines:

>>  integer, intent(IN) :: N
>>  integer :: M(N,N)

If this was C, such a declaration would mean a variable-length array
that would have to be placed on the (limited-size) stack and eventually
overflow it. gfortran places the array on the heap, so the program
works:

  integer, intent(IN) :: N
  integer, intent(INOUT) :: V(N*N)
  integer :: M(N,N)
    1205:       48 63 db                movslq %ebx,%rbx
    1208:       b8 00 00 00 00          mov    $0x0,%eax
    120d:       48 85 db                test   %rbx,%rbx
    1210:       49 89 c4                mov    %rax,%r12
    1213:       4c 0f 49 e3             cmovns %rbx,%r12
    1217:       48 89 df                mov    %rbx,%rdi
    121a:       49 0f af fc             imul   %r12,%rdi
    121e:       48 85 ff                test   %rdi,%rdi
    1221:       48 0f 48 f8             cmovs  %rax,%rdi
    1225:       48 c1 e7 02             shl    $0x2,%rdi
    1229:       b8 01 00 00 00          mov    $0x1,%eax
    122e:       48 0f 44 f8             cmove  %rax,%rdi
    1232:       e8 19 fe ff ff          callq  1050 <malloc using plt>
    1237:       48 89 c5                mov    %rax,%rbp
    123a:       4c 89 e7                mov    %r12,%rdi
    123d:       48 f7 d7                not    %rdi

(Looking at the address of M in GDB and comparing it with the output
of info proc mappings, I can confirm that it lives on the heap.)

flang-new makes M into a C-style VLA:

  integer, intent(IN) :: N
  integer, intent(INOUT) :: V(N*N)
  integer :: M(N,N)
    74ec:       48 63 17                movslq (%rdi),%rdx
    74ef:       89 d1                   mov    %edx,%ecx
    74f1:       31 c0                   xor    %eax,%eax
    74f3:       48 85 d2                test   %rdx,%rdx
    74f6:       48 0f 49 c2             cmovns %rdx,%rax
    74fa:       48 89 85 b0 fe ff ff    mov    %rax,-0x150(%rbp)
    7501:       48 89 c2                mov    %rax,%rdx
    7504:       48 0f af d2             imul   %rdx,%rdx
    7508:       48 8d 34 95 0f 00 00    lea    0xf(,%rdx,4),%rsi
    750f:       00
    7510:       48 83 e6 f0             and    $0xfffffffffffffff0,%rsi
    7514:       48 89 e2                mov    %rsp,%rdx
    7517:       48 29 f2                sub    %rsi,%rdx
    751a:       48 89 95 b8 fe ff ff    mov    %rdx,-0x148(%rbp)
    7521:       48 89 d4                mov    %rdx,%rsp

(Looking at the value of the stack pointer in GDB after M(N,N) is
declared, I can see it way below the end of the stack and the loaded
shared libraries according to info proc mappings. GDB doesn't let me
see the address of M. The program crashes in `M = 42`, trying to
overwrite the code from the C standard library.)

Are Fortran processors allowed to place such "automatic data objects"
like integer :: M(N,N) on the stack? The Fortran standard doesn't seem
to give an answer to this question, but if you make your M allocatable,
you won't have to worry about stack usage:

subroutine dostuff(N,V)
  implicit none

  integer, intent(IN) :: N
  integer, intent(INOUT) :: V(N*N)
  integer, allocatable :: M(:,:) ! <-- here

  allocate(M(N,N))               ! <-- and here
  M = 42
  V = RESHAPE(M, (/N*N/))
end subroutine dostuff

No leaks or crashes observed with these two changes and either
compiler. The Fortran standard requires that local allocatable unsaved
arrays (except for the function result) are deallocated at the end of
procedures.

-- 
Best regards,
Ivan