[R] Reading large files with R
murdoch@dunc@n @end|ng |rom gm@||@com
Sun Sep 1 21:53:55 CEST 2019
On 01/09/2019 3:06 p.m., Martin Møller Skarbiniks Pedersen wrote:
> I am trying to read yaml-file which is not so large (7 GB) and I have
> plenty of memory.
> However I get this error:
> $ R --version
> R version 3.6.1 (2019-07-05) -- "Action of the Toes"
> Copyright (C) 2019 The R Foundation for Statistical Computing
> Platform: x86_64-pc-linux-gnu (64-bit)
> keys <- read_yaml("/data/gpg/gpg-keys.yaml")
> Error in paste(readLines(file), collapse = "\n") :
> result would exceed 2^31-1 bytes
> 2^31-1 is only 2GB.
> Please advise,
Individual elements in character vectors have a size limit of 2^31-1.
The read_yaml() function is putting the whole file into one element, and
You probably have a couple of choices:
- Rewrite read_yaml() so it doesn't try to do that. This is likely
hard, because most of the work is being done by a C routine, but it's
conceivable you could use the stringi::stri_read_raw function to do the
reading, and convince the C routine to handle the raw value instead of a
- Find a way to split up your file into smaller pieces.
More information about the R-help