[R] compress data on read, decompress on write

[Ramon Diaz-Uriarte]

Thanks, Greg. Yes, I'd store the compressed stuff as a raw data type.

Best,

R.

On Thu, Feb 28, 2008 at 11:54 PM, Gregory Warnes <gregory.warnes at mac.com> wrote:
>
>  You might look at storing the data using R's "raw" data type...
>
>  -G
>
>
>
>
>  On Feb 28, 2008, at 5:38PM , Ramon Diaz-Uriarte wrote:
>
>  > Dear Christos,
>  >
>  > Thanks for your reply. Actually, I should have been more careful with
>  > language: its not really a sparse matrix, but rather a ragged array
>  > that results from a more compact representation we though of for the
>  > hidden states in a Hidden Markov Model in many runs of MCMC. However,
>  > it might make sense for us to check sparseMatrix and see how its done
>  > there.
>  >
>  > Thanks,
>  >
>  > R
>  >
>  > On Thu, Feb 28, 2008 at 7:49 PM, Christos Hatzis
>  > <christos.hatzis at nuverabio.com> wrote:
>  >> Ramon,
>  >>
>  >>  If you are looking for a solution to your specific application
>  >> (as opposed
>  >>  to a general compression/ decompression mechanism), it might be
>  >> worth
>  >>  checking out the Matrix package, which has facilities for storing
>  >> and
>  >>  manipulating sparse matrices.  The sparseMatrix class stores
>  >> matrices in the
>  >>  triplet representation (i.e. only indices and values of the non-zero
>  >>  elements) and this affords great compression ratios, depending on
>  >> the size
>  >>  and degree of sparseness of the matrix.
>  >>
>  >>  -Christos
>  >>
>  >>
>  >>
>  >>> -----Original Message-----
>  >>> From: r-help-bounces at r-project.org
>  >>> [mailto:r-help-bounces at r-project.org] On Behalf Of Ramon Diaz-
>  >>> Uriarte
>  >>> Sent: Thursday, February 28, 2008 1:18 PM
>  >>> To: r-help at stat.math.ethz.ch
>  >>> Subject: [R] compress data on read, decompress on write
>  >>>
>  >>> Dear All,
>  >>>
>  >>> I'd like to be able to have R store (in a list component) a
>  >>> compressed data set, and then write it out uncompressed.
>  >>> gzcon and gzfile work in exactly the opposite direction. What
>  >>> would be a good way to handle this?
>  >>>
>  >>> Details:
>  >>> ----------
>  >>>
>  >>> We have a package that uses C; part of the C output is a
>  >>> large sparse matrix. This is never manipulated directly by R,
>  >>> but always by the C code. However, we need to store that data
>  >>> somewhere (inside an R
>  >>> object) for further calls to the functions in our package.
>  >>> We'd like to store that matrix as part of the R object (say,
>  >>> as an element of a list). Ideally, it would be stored in as
>  >>> compressed a way as possible.
>  >>> Then, when we need to use that information, it would be
>  >>> decompressed and passed to the C function.
>  >>>
>  >>> I guess one way to do it is to have C deal with the
>  >>> compression and uncompression (e.g., using zlib or the bzip2
>  >>> libraries) and then use readBin, etc, from R. But, if I can,
>  >>> I'd like to avoid our C code having to call zlib, etc, so as
>  >>> to make our package easily portable.
>  >>>
>  >>>
>  >>> Thanks,
>  >>>
>  >>> R.
>  >>>
>  >>> --
>  >>> Ramon Diaz-Uriarte
>  >>> Statistical Computing Team
>  >>> Structural Biology and Biocomputing Programme Spanish
>  >>> National Cancer Centre (CNIO) http://ligarto.org/rdiaz
>  >>>
>  >>> ______________________________________________
>  >>> R-help at r-project.org mailing list
>  >>> https://stat.ethz.ch/mailman/listinfo/r-help
>  >>> PLEASE do read the posting guide
>  >>> http://www.R-project.org/posting-guide.html
>  >>> and provide commented, minimal, self-contained, reproducible code.
>  >>>
>  >>>
>  >>
>  >>
>  >>
>  >
>  >
>  >
>  > --
>  > Ramon Diaz-Uriarte
>  > Statistical Computing Team
>  > Structural Biology and Biocomputing Programme
>  > Spanish National Cancer Centre (CNIO)
>  > http://ligarto.org/rdiaz
>  >
>  > ______________________________________________
>  > R-help at r-project.org mailing list
>  > https://stat.ethz.ch/mailman/listinfo/r-help
>  > PLEASE do read the posting guide http://www.R-project.org/posting-
>  > guide.html
>  > and provide commented, minimal, self-contained, reproducible code.
>
>



-- 
Ramon Diaz-Uriarte
Statistical Computing Team
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO)
http://ligarto.org/rdiaz