[R-pkg-devel] libarrow

[Dewey Dunnington]

I'll also add that you can write R level code that processes files 
lazily chunk-by-chunk
using arrow::open_dataset() |> arrow::map_batches(), and that you can 
write
C level code that processes batches chunk-by-chunk using 
arrow::open_dataset() |>
nanoarrow::as_nanoarrow_array_stream(), which will give you a pointer to
an ABI stable ArrowArrayStream that you can pass between packages 
(helpers
exist in the Arrow rust implementation and the nanoarrow C/C++ library 
to help
consume that, and I'm happy to work up an example if you are 
interested).

Cheers,

-dewey

On 2024-11-26 15:45, Bryce Mecum wrote:
> The work in progress to add libarrow to MXE is located at [1]. I don't
> think it's complete but I'm sharing it here in case it's a useful
> starting point for someone. Continuing this work would be welcome.
> 
> Thang: Feel free to file an issue on the Arrow issue tracker [2] or
> email the user@ mailing list [3] if you get stuck.
> 
> [1] https://github.com/assignUser/cran-mxe
> [2] https://github.com/apache/arrow/issues
> [3] https://arrow.apache.org/community/
> 
> On Tue, Nov 26, 2024 at 3:11 AM Tomas Kalibera 
> <tomas.kalibera using gmail.com> wrote:
>> 
>> On 11/26/24 11:18, Pham, T.V. via R-package-devel wrote:
>> > Dear all,
>> >
>> > I am planning to use libarrow for reading parquet files, chunk by chunk. It seems that by installing "arrow", one gets the libarrow library too (at least on Windows).
>> 
>> On Windows, to properly use Apache Arrow with R, it would have to be
>> added to Rtools (and hence ideally first upstream to MXE). I've been 
>> in
>> touch with two arrow developers who started working on adding libarrow
>> to MXE, but then ran out of time. If there were any volunteers to 
>> finish
>> this, perhaps its best to check first with the arrow team where they
>> got, rather than starting from scratch.
>> 
>> Best
>> Tomas
>> 
>> >
>> > I wonder if the C++ header files are already in some package like Rcpp. Also, how should the link to the libarrow look like?
>> >
>> > Thank you & best regards,
>> > Thang
>> >
>> > ______________________________________________________
>> > AmsterdamUMC disclaimer : www.amsterdamumc.org/nl/disclaimers.htm
>> >
>> > ______________________________________________
>> > R-package-devel using r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> 
>> ______________________________________________
>> R-package-devel using r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> 
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel