[R-pkg-devel] [EXTERNAL] Re: Fast Matrix Serialization in R?

[Sameh Abdulah]

Thanks!


I want to save data from a matrix to a file and then retrieve it later while running R code, all within R.

As long as the compression doesn't result in data loss, it should be suitable for my needs.


Best,
--Sameh



From: Dirk Eddelbuettel <edd using debian.org>
Date: Thursday, May 9, 2024 at 6:35 AM
To: Sameh Abdulah <sameh.abdulah using kaust.edu.sa>
Cc: r-package-devel using r-project.org <r-package-devel using r-project.org>
Subject: [EXTERNAL] Re: [R-pkg-devel] Fast Matrix Serialization in R?

On 9 May 2024 at 03:20, Sameh Abdulah wrote:
| I need to serialize and save a 20K x 20K matrix as a binary file.

Hm that is an incomplete specification: _what_ do you want to do with it?
Read it back in R?  Share it with other languages (like Python) ? I.e. what
really is your use case?  Also, you only seem to use readBin / writeBin. Why
not readRDS / saveRDS which at least give you compression?

If it is to read/write from / to R look into the qs package. It is good. The
README.md at its repo has benchmarks: https://urldefense.com/v3/__https://github.com/traversc/qs__;!!Nmw4Hv0!zZitnMd5aMqDWhLShhC5Npmd8pkisFVejC3grX2YGSGWFZihVaHAezYyo5nXbAwIdqCj2T8Nbk-Yofxa-y8$<https://urldefense.com/v3/__https:/github.com/traversc/qs__;!!Nmw4Hv0!zZitnMd5aMqDWhLShhC5Npmd8pkisFVejC3grX2YGSGWFZihVaHAezYyo5nXbAwIdqCj2T8Nbk-Yofxa-y8$>  If you
want to index into the stored data look into fst. Else also look at databases

Dirk

--
dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org

-- 

This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.

	[[alternative HTML version deleted]]