[Rd] Large vector support in data.frames

[Simon Urbanek]

The second point is not really an issue - R already uses numerics for larger-than-32-bit indexing at R level and it works just fine for objects up to ca. 72 petabytes.

However, the first one is a bit more relevant than one would think. At one point I have experimented with allowing data frames with more than 2^31 rows, but it breaks in many places - some quite unexpected. Beside dim() there is also the issue with (non-expanded) row names. Overall, it is a lot more work - some would have to be done in R but some would require changes to packages as well.

(In practice I use sharded data frames for large data which removes the limit and allows parallel processing - but requires support from the methods that will be applied to them).

Cheers,
Simon



> On Jul 2, 2024, at 16:04, Ivan Krylov via R-devel <r-devel using r-project.org> wrote:
> 
> В Wed, 19 Jun 2024 09:52:20 +0200
> Jan van der Laan <rhelp using eoos.dds.nl> пишет:
> 
>> What is the status of supporting long vectors in data.frames (e.g. 
>> data.frames with more than 2^31 records)? Is this something that is 
>> being worked on? Is there a time line for this? Is this something I
>> can contribute to?
> 
> Apologies if you've already received a better answer off-list.
> 
> From from my limited understanding, the problem with supporting
> larger-than-(2^31-1) dimensions has multiple facets:
> 
> - In many parts of R code, there's the assumption that dim() is
>   of integer type. That wouldn't be a problem by itself, except...
> 
> - R currently lacks a native 64-bit integer type. About a year ago
>   Gabe Becker mentioned that Luke Tierney has been considering
>   improvements in this direction, but it's hard to introduce 64-bit
>   integers without making the user worry even more about data types
>   (numeric != integer != 64-bit integer) or introducing a lot of
>   overhead (64-bit integers being twice as large as 32-bit ones and,
>   depending on the workload, frequently redundant).
> 
> - Two-dimensional objects eventually get transformed into matrices and
>   handed to LAPACK for linear algebra operations. Currently, the
>   interface used by R to talk to BLAS and LAPACK only supports 32-bit
>   signed integers for lengths. 64-bit BLASes and LAPACKs do exist
>   (e.g. OpenBLAS can be compiled with 64-bit lengths), but we haven't
>   taught R to use them.
> 
>   (This isn't limited to array dimensions, by the way. If you try to
>   svd() a 40000 by 40000 matrix, it'll try to ask for temporary memory
>   with length that overflows a signed 32-bit integer, get a much
>   shorter allocation instead, promptly overflow the buffer and
>   crash the process.)
> 
> As you see, it's interconnected; work on one thing will involve the
> other two.
> 
> -- 
> Best regards,
> Ivan
> 
> ______________________________________________
> R-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>