[Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648 elements

[Steven McKinney]

Hi Colton,

You could divide your write task into chunks that do not violate the 2^31-1 limit.

write.table has an append argument (default FALSE). 

Figure out a row chunk size nri < nr such that nri * nc is under 2^31-1 and use
write.table() to write that out.

Then use 
write.table(  append = TRUE, )
for the next chunk of rows, looping over chunks until done.  Two chunks will get your 2.8 billion entries done.


Best

Steve



Steven McKinney, Ph.D.

Statistician
Molecular Oncology and Breast Cancer Program
British Columbia Cancer Research Centre




> -----Original Message-----
> From: R-devel [mailto:r-devel-bounces at r-project.org] On Behalf Of Tousey,
> Colton
> Sent: April-18-18 2:08 PM
> To: R-core at r-project.org; simon.urbanek at R-project.org; R-devel at r-
> project.org
> Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648
> elements
> 
> Hello,
> 
> I want to report a bug in R that is limiting my capabilities to export a
> matrix with write.csv or write.table with over 2,147,483,648 elements (C's
> int limit). I found this bug already reported about before: https://bugs.r-
> project.org/bugzilla/show_bug.cgi?id=17182. However, there appears to be no
> solution or fixes in upcoming R version releases.
> 
> The error message is coming from the writetable part of the utils package
> in the io.c source code(https://svn.r-
> project.org/R/trunk/src/library/utils/src/io.c):
> /* quick integrity check */
>                 if(XLENGTH(x) != (R_len_t)nr * nc)
>                     error(_("corrupt matrix -- dims not not match
> length"));
> 
> The issue is that nr*nc is an integer and the size of my matrix, 2.8
> billion elements, exceeds C's limit, so the check forces the code to fail.
> 
> My version:
> > R.Version()
> $platform
> [1] "x86_64-w64-mingw32"
> 
> $arch
> [1] "x86_64"
> 
> $os
> [1] "mingw32"
> 
> $system
> [1] "x86_64, mingw32"
> 
> $status
> [1] ""
> 
> $major
> [1] "3"
> 
> $minor
> [1] "4.3"
> 
> $year
> [1] "2017"
> 
> $month
> [1] "11"
> 
> $day
> [1] "30"
> 
> $`svn rev`
> [1] "73796"
> 
> $language
> [1] "R"
> 
> $version.string
> [1] "R version 3.4.3 (2017-11-30)"
> 
> $nickname
> [1] "Kite-Eating Tree"
> 
> Thank you,
> Colton
> 
> 
> Colton Tousey
> Research Associate II
> P: 816.585.0300   E: colton.tousey at kc.frb.org
> FEDERAL RESERVE BANK OF KANSAS CITY
> 1 Memorial Drive   *   Kansas City, Missouri 64198   *
> www.kansascityfed.org
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel