[R] Reading large files with R

[Duncan Murdoch]

On 01/09/2019 3:06 p.m., Martin Møller Skarbiniks Pedersen wrote:
> Hi,
> 
>    I am trying to read yaml-file which is not so large (7 GB) and I have
> plenty of memory.
> However I get this error:
> 
> $  R --version
> R version 3.6.1 (2019-07-05) -- "Action of the Toes"
> Copyright (C) 2019 The R Foundation for Statistical Computing
> Platform: x86_64-pc-linux-gnu (64-bit)
> 
> library(yaml)
> keys <- read_yaml("/data/gpg/gpg-keys.yaml")
> 
> Error in paste(readLines(file), collapse = "\n") :
>    result would exceed 2^31-1 bytes
> 
> 2^31-1 is only 2GB.
> 
> Please advise,
> 
> Regards
> Martin

Individual elements in character vectors have a size limit of 2^31-1. 
The read_yaml() function is putting the whole file into one element, and 
that's failing.

You probably have a couple of choices:

  - Rewrite read_yaml() so it doesn't try to do that.  This is likely 
hard, because most of the work is being done by a C routine, but it's 
conceivable you could use the stringi::stri_read_raw function to do the 
reading, and convince the C routine to handle the raw value instead of a 
character value.

  - Find a way to split up your file into smaller pieces.

Duncan Murdoch