[R] mmap error-12, macbook pro
tlumley at u.washington.edu
Wed Mar 17 17:34:20 CET 2010
There seem to have been three almost identical reports of this from three different email addresses, including two widely-separate UK academic institutions, and two user names, which is a bit weird.
However: R will need more than 4GB memory to handle a 3.99GB file, and probably more than 4GB memory to handle a 1.33GB file.
On 32-bit R I think the recommendation is that the data should occupy no more than 10% of the address space, ie, about 400MB. For 64-bit R, you are likely to still find R very slow if the data set is more than about 1/3 of physical memory. For a 4GB data file I would recommend a computer with at least 16GB memory.
Now, it is often possible to load only a small fraction of the data at one time and so to analyse large data sets on smaller computers. Three examples that I have worked on
- analysing data from a survey data set about the same size as your data, by keeping most of the data in a SQLite database and just loading a
few variables at a time.
- fitting linear regression models to large data sets with the biglm package, by keeping the data in a SQLite database and just loading a few rows
at a time
- analysing whole-genome genetic data (40GB or so) by storing the data in a netCDF file and reading appropriate chunks with the ncdf package.
On Wed, 17 Mar 2010, Mamun wrote:
> Dear List,
> I am trying to read some files using read.csv and total size of those files
> are 3.99 GB. I am using MacBook Pro with 4GB RAM(snow leopard). I also tried
> to run a chunk from those files and altogether the size was 1.33 GB. But
> every time I was getting the following error
> R(1200) malloc: *** mmap(size=16777216) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> I managed to run smaller chunks (400MB) and I also saved the object. But
> problem is when I try to load some of those objects(which is not more than
> 1.5 GB altogether), I get same error again. Can someone please help? why am
> I getting this error? does R need more space than the actual file size? I
> may buy new machine if its something related to RAM size. but if it is some
> R problem then it will be very useless to buy new machine. So, I really need
> to know why this is happening.
> Any help is appreciated.
> Thanks in advance.
> View this message in context: http://n4.nabble.com/mmap-error-12-macbook-pro-tp1596234p1596234.html
> Sent from the R help mailing list archive at Nabble.com.
> R-help at r-project.org mailing list
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
Thomas Lumley Assoc. Professor, Biostatistics
tlumley at u.washington.edu University of Washington, Seattle
More information about the R-help