[Rd] R process killed when allocating too large matrix (Mac OS X)

[Martin Maechler]

>>>>> Kirill Müller <kirill.mueller at ivt.baug.ethz.ch>
>>>>>     on Wed, 11 May 2016 10:42:56 +0200 writes:

    > My ulimit package exposes this API ([1], should finally submit it to 
    > CRAN); unfortunately this very API seems to be unsupported on OS X 
    > [2,3]. Last time I looked into it, neither of the documented settings 
    > achieved the desired effect.

    > -Kirill

    > [1] http://krlmlr.github.io/ulimit
    > [2] 
    > http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
    > [3] 
    > https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


    > On 10.05.2016 01:08, Jeroen Ooms wrote:
    >> On 05/05/2016 10:11, Uwe Ligges wrote:
    >>> Actually this also happens under Linux and I had my R processes killed
    >>> more than once (and much worse also other processes so that we had to
    >>> reboot a server, essentially).

I agree that Linux is not consistently fine here either.

    >> I found that setting RLIMIT_AS [1] works very well on Linux. But this
    >> requires that you cap memory to some fixed value.

conceivably one could set a default cap, using something
equivalent to the data in

		sfsmisc::Sys.meminfo()
or		sfsmisc::Sys.memGB()		

(very simple, > 10 year old interfaces, based on the  Linux-only (?)
 '/proc/*' filesystem).


In an ideal word, some of us,
    from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
    maintainer("microbenchmark>, ...
would sit together and devise an R function interface (based on
low level platform specific interfaces, specifically for at least
Linux/POSIX-compliant, Mac, and Windows) which would allow
something  like your rlimit(..) calls below.

We'd really need something to work on all platforms ideally,
to be used by R package maintainers
and possibly even better by R itself at startup, setting a
reasonable memory cap - which the user could raise even to +Inf (or lower
even more).

Martin

---------
Final notes about RAppArmor, not relevant to the main thread topic :

Note: I'm working in pretty well maintained Fedora Linux env,
but Apparmor is not only not activated, but even not available.
OTOH, usting  RLIMIT / gerlimit on Linux is very generally
available.

As a consequence, the three last lines of

   > require(RAppArmor)
   Loading required package: RAppArmor
   Loading required package: tools
   Failed to lookup process confinement:
   AppArmor not supported on this system
   Have a look at: sudo aa-status

were very confusing to me:  My conclusion was I could not use
the RAppArmor package.

(But that's wrong: For the  rlimit*()  functions below, one do
 *NOT* need an AppArmor-enabled version of Linux !)
		  
    >>> library(RAppArmor)
    >>> rlimit_as(1e9)
    >>> rnorm(1e9)
    >> Error: cannot allocate vector of size 7.5 Gb
    >> 
    >> The RAppArmor package has many other utilities to protect your server
    >> such from a mis-behaving process such as limiting cpu time
    >> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).
    >> 
    >> [1] http://linux.die.net/man/2/getrlimit

and from my current explorations I gather that all of these are
*not* Apparmor related... so could/should maybe rather migrate
into a lightweight package not mentioning AppArmor ?