[Rd] R process killed when allocating too large matrix (Mac OS X)

[Simon Urbanek]

On May 4, 2016, at 9:00 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote:

> Hi Simon,
> 
> thanks for your quick reply.
> 
> 1) ... so you can reproduce this?

Yes, I can on 10.11.4.


> 2) Do you know a way how this can be 'foreseen'? We allocate larger
> matrices in the copula package depending on the user's input
> dimension. It would be good to tell her/him "Your dimension is quite
> large. Be aware of killers in your neighborhood"... before the killer
> attacks.
> 

Not directly, because the system is happy to accommodate R's request for more memory but then it will strike out of the blue. Since that decision is made by the kernel and it doesn't expose its thinking about how it feels, it's hard to tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, unfortunately.

You could guess - e.g. by checking the total system memory

int mib [] = { CTL_HW, HW_MEMSIZE };
int64_t value = 0;
size_t length = sizeof(value);
sysctl(mib, 2, &value, &length, NULL, 0);

and comparing it to the sizes involved. However, even that is not foolproof, because it all depends on the other processes' memory usage, swap space size etc. There are slightly more involved ways to query the VM system as well, but I'm not sure I'd want to go so deep into the weeds, especially since this becomes quickly highly OS-specific.

Cheers,
Simon



> Thanks & cheers,
> Marius
> 
> 
> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
> <simon.urbanek at r-project.org> wrote:
>> 
>> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote:
>> 
>>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).
>>> 
>>> Hi Simon!
>>> 
>>> Sure, but who do you tell it if you don't know the killer?
>>> This is all the killer left me with, the 'crime scene' if you like :-)
>>> 
>>>> m <- matrix(0, 90000, 100000)
>>> Killed: 9
>>> 
>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>> apparently the killer went further down the hallway to his office
>>> now... so scary. Here is Wayne's sessionInfo():
>>> 
>> 
>> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see
>> 
>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>> 
>> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning.
>> 
>> Cheers,
>> Simon
>> 
>