[R] Error: cons memory exhausted (limit reached?): Memory Management?

Christopher Lloyd chr|@||oyd2 @end|ng |rom y@hoo@co@uk
Tue Jan 19 21:22:07 CET 2021


Hi all, I think this is only the second time that I have posted so I apologise if my etiquette isn't quite correct.
I'm loading a large (~30GB) geojson file into R using readOGR on a HPC. I am also loading a small shapefile, and then trying to undertake some processing on the large geojson using gBuffer from the rgeos package.

I believe that the HPC is running Red Hat Enterprise Linux 7.4, and it certainly has around 750 GB free for user jobs. I have allocated the full amount of ram to the job.
I previously used the following modules to undertake this task and it ran successfully, although only after tweaking the settings that I detail below - otherwise I had the same error:

module load proj/5.0.0
module load gdal/2.3.1
module load geos/3.6.2
module load gcc/6.4.0
module load R/3.5.2
module load python  # python 3 by default
module load numpy/1.14.0 # requires module load python

Settings at linux command line that previously allowed a successful run:
R_MAX_VSIZE=720G
R_GC_MEM_GROW=0

--min-nsize=50000k --min-vsize=12M --max-ppsize=500000 (when executing the R script from command line)

However, the modules have now been updated on the HPC, and so I am now using:
module load proj/6.1.1
module load R/3.6.2
(other modules remain the same)

I get the following error whilst processing (loading the file into R is ok), with gcinfo() turned on:

Garbage collection 144 = 86+22+36 (level 0) ... 
288541.3 Mbytes of cons cells used (66%)
55320.1 Mbytes of vectors used (98%)
Garbage collection 145 = 86+23+36 (level 1) ... 
66679.3 Mbytes of cons cells used (15%)
56447.7 Mbytes of vectors used (100%)
Garbage collection 146 = 86+23+37 (level 2) ... 
39852.3 Mbytes of cons cells used (11%)
49032.4 Mbytes of vectors used (72%)
Garbage collection 147 = 87+23+37 (level 0) ... 
124935.0 Mbytes of cons cells used (36%)
64961.0 Mbytes of vectors used (95%)
Garbage collection 148 = 87+24+37 (level 1) ... 
985162418403226.2 Mbytes of cons cells used (-2147483648%)
35274.8 Mbytes of vectors used (52%)

Error: cons memory exhausted (limit reached?)
In addition: Warning message:
Garbage collection 149 = 88+24+37 (level 0) ... 
985162418403226.2 Mbytes of cons cells used (-2147483648%)
35274.8 Mbytes of vectors used (52%)
Lost warning messages
Execution halted
Garbage collection 150 = 89+24+37 (level 0) ... 
985162418403226.2 Mbytes of cons cells used (-2147483648%)
35274.8 Mbytes of vectors used (52%)

Error: cons memory exhausted (limit reached?)

The job halts with Memory Utilized: 411.29 GB

I cannot understand why the job worked previously (just) but now does not when seemingly the only change is an updated proj and R version (3.5.2 to 3.6.2).

Might anyone have any suggestions as to why this is the case? And/or how to alter the memory management so that memory is not exhausted so easily?

Many thanks, Chris

	[[alternative HTML version deleted]]



More information about the R-help mailing list