[R] Memory limit on Linux?

[Stackpole, Chris]

> From: Kevin E. Thorpe [mailto:kevin.thorpe at utoronto.ca] 
> Sent: Tuesday, August 13, 2013 2:25 PM
> Subject: Re: [R] Memory limit on Linux?
>
> It appears that at the shell level, the differences are not to blame. 
> It has been a long time, but years ago in HP-UX, we needed to change an
> actual kernel parameter (this was for S-Plus 5 rather than R back then). 
> Despite the ulimits being acceptable, there was a hard limit in the kernel.
> I don't know whether such things have been (or can be) built in to your
> "problem" machine.  If it is a multiuser box, it could be that limits have
> been set to prevent a user from gobbling up all the memory.

I thought about that too as I was not the admin who built the box (I took over for him), but I don't see anything at all in the kernel, climits, or anything else that suggests this is the case.

> The other thing to check is if R has/can be compiled with memory limits.

That was the second question of my original post. I looked but I don't see anything related to this except for various posts dealing with a windows box.

> Sorry I can't be of more help.

No worries. I do appreciate your help.

I finally got access to the code and data set for my own testing so I did some more research while running a job. I looked around in the /proc/$taskid folder [on the box that seems to stop using memory around 5GB] but nothing really jumped out at me. Maybe someone else will catch something I missed.
$ cat limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            unlimited            unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             1024                 2066361              processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       2066361              2066361              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

$ cat statm
1347444 1312394 752 1 0 1312166 0
Memory Size=1347444~=5263M
Resident memory=1312394~=5126M
Libraries=1312166~=5125M

$cat status
Name:	R
State:	R (running)
Tgid:	23015
Pid:	23015
PPid:	23011
TracerPid:	0
Uid:	1701	1701	1701	1701
Gid:	680	680	680	680
Utrace:	0
FDSize:	256
Groups:	680 
VmPeak:	 5389776 kB
VmSize:	 5358464 kB
VmLck:	       0 kB
VmHWM:	 5249576 kB
VmRSS:	 5218420 kB
VmData:	 5217160 kB
VmStk:	     192 kB
VmExe:	       4 kB
VmLib:	    9200 kB
VmPTE:	   10352 kB
VmSwap:	       0 kB
Threads:	1
SigQ:	0/2066361
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	0000000180001e4a
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	ffffffffffffffff
Cpus_allowed:	ffffffff
Cpus_allowed_list:	0-31
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f
Mems_allowed_list:	0-3
voluntary_ctxt_switches:	98
nonvoluntary_ctxt_switches:	101913


On the system in which more memory allocates freely (this particular job isn't quite as big as the others, so ~10GB in size is correct):

$ cat limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             773356               773356               processes 
Max open files            1024                 1024                 files     
Max locked memory         32768                32768                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       773356               773356               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    

$ cat statm
2746521 2716426 841 1 0 2716928 0
Memory Size=2746521~=10728M
Resident memory=2716426~=10611M
Libraries=2716928~=10613M

$ cat status
Name:	R
State:	R (running)
SleepAVG:	0%
Tgid:	17003
Pid:	17003
PPid:	16984
TracerPid:	0
Uid:	46877	46877	46877	46877
Gid:	25043	25043	25043	25043
FDSize:	256
Groups:	351 25043 48628 59110 
VmPeak:	11109136 kB
VmSize:	10986084 kB
VmLck:	       0 kB
VmHWM:	10988628 kB
VmRSS:	10865704 kB
VmData:	10867572 kB
VmStk:	     140 kB
VmExe:	       4 kB
VmLib:	   19892 kB
VmPTE:	   21396 kB
StaBrk:	1f1a5000 kB
Brk:	1b310b000 kB
StaStk:	7fff0ddf6ba0 kB
Threads:	1
SigQ:	1/773356
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	0000000180001e4a
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
Cpus_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00ffffff
Mems_allowed:	00000000,00000003

Thanks!
Chris Stackpole