[Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648 elements

[Spencer Graves]

On 2018-04-18 17:38, Steven McKinney wrote:
> Hi Colton,
>
> You could divide your write task into chunks that do not violate the 2^31-1 limit.
>
> write.table has an append argument (default FALSE).
>
> Figure out a row chunk size nri < nr such that nri * nc is under 2^31-1 and use
> write.table() to write that out.
>
> Then use
> write.table(  append = TRUE, )
> for the next chunk of rows, looping over chunks until done.  Two chunks will get your 2.8 billion entries done.


       Magnificent:  Is that something that could be implemented inside 
write.table?


       Spencer
>
> Best
>
> Steve
>
>
>
> Steven McKinney, Ph.D.
>
> Statistician
> Molecular Oncology and Breast Cancer Program
> British Columbia Cancer Research Centre
>
>
>
>
>> -----Original Message-----
>> From: R-devel [mailto:r-devel-bounces at r-project.org] On Behalf Of Tousey,
>> Colton
>> Sent: April-18-18 2:08 PM
>> To: R-core at r-project.org; simon.urbanek at R-project.org; R-devel at r-
>> project.org
>> Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648
>> elements
>>
>> Hello,
>>
>> I want to report a bug in R that is limiting my capabilities to export a
>> matrix with write.csv or write.table with over 2,147,483,648 elements (C's
>> int limit). I found this bug already reported about before: https://bugs.r-
>> project.org/bugzilla/show_bug.cgi?id=17182. However, there appears to be no
>> solution or fixes in upcoming R version releases.
>>
>> The error message is coming from the writetable part of the utils package
>> in the io.c source code(https://svn.r-
>> project.org/R/trunk/src/library/utils/src/io.c):
>> /* quick integrity check */
>>                  if(XLENGTH(x) != (R_len_t)nr * nc)
>>                      error(_("corrupt matrix -- dims not not match
>> length"));
>>
>> The issue is that nr*nc is an integer and the size of my matrix, 2.8
>> billion elements, exceeds C's limit, so the check forces the code to fail.
>>
>> My version:
>>> R.Version()
>> $platform
>> [1] "x86_64-w64-mingw32"
>>
>> $arch
>> [1] "x86_64"
>>
>> $os
>> [1] "mingw32"
>>
>> $system
>> [1] "x86_64, mingw32"
>>
>> $status
>> [1] ""
>>
>> $major
>> [1] "3"
>>
>> $minor
>> [1] "4.3"
>>
>> $year
>> [1] "2017"
>>
>> $month
>> [1] "11"
>>
>> $day
>> [1] "30"
>>
>> $`svn rev`
>> [1] "73796"
>>
>> $language
>> [1] "R"
>>
>> $version.string
>> [1] "R version 3.4.3 (2017-11-30)"
>>
>> $nickname
>> [1] "Kite-Eating Tree"
>>
>> Thank you,
>> Colton
>>
>>
>> Colton Tousey
>> Research Associate II
>> P: 816.585.0300   E: colton.tousey at kc.frb.org
>> FEDERAL RESERVE BANK OF KANSAS CITY
>> 1 Memorial Drive   *   Kansas City, Missouri 64198   *
>> www.kansascityfed.org
>>
>> 	[[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel