[R] Dec. 1, 2009 tip of the day

[John Christie]

RE: Compression

Hi R-Users,

You can deal with pretty decent size data sets in R on a relatively new computer.  I have one that I have been working with that is a nearly 100MB plain text file.  With storage as inexpensive as it is these days that's not really all that much data and I could store it just as it is.

Having said that you may want to compress those data files.  There are two reasons for this.  One is that while storage is cheap, large files can be harder to move around.  Emailing is an example of something that's difficult to do with a larger data file.  The other is that, even though your drive has lots of capacity and seems to go pretty fast, it is in fact much much slower than your CPU.  Recognizing this, Apple built transparent background compression into the file system in it's latest operating system.  It speeds things up because the computer spends less time accessing the disk (and more time using the CPU decompressing and compressing it, but the CPU usually wasn't doing anything anyway, and remember, it's much faster).

It turns out R added transparent decompression for certain kinds of compressed files in the latest version (2.10).  If you have your files compressed with bzip2, xvz, or gzip they can be read into R as if they are plain text files.  You should have the proper filename extensions.

The command...

myData <- read.table('myFile.gz')  #gzip compressed files have a "gz" extension

Will work just as if 'myFile.gz' were the raw text file.

This is very handy in distributing your analysis scripts and data to co-workers at the same time and saves you space on your hard drive.  More importantly, it saves you space on flash drives and backups.

John

PS:  My compressed 100MB data file is 500KB.  That's tiny these days.  Programs to do the kinds of compression acceptable for R are available for free.  If you have a Mac or Linux computer they are built into your command line and are as simple as typing gzip 'filename'.

bonus tip:  R has built in facilities for writing out the compressed files as well.  "?connections" gets you the help page with basic info.