[R] Reasons to Use R

[Wensui Liu]

Greg,
As far as I understand, SAS is more efficient handling large data
probably than S+/R. Do you have any idea why?

On 4/10/07, Greg Snow <Greg.Snow at intermountainmail.org> wrote:
> > -----Original Message-----
> > From: r-help-bounces at stat.math.ethz.ch
> > [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of
> > Bi-Info (http://members.home.nl/bi-info)
> > Sent: Monday, April 09, 2007 4:23 PM
> > To: Gabor Grothendieck
> > Cc: Lorenzo Isella; r-help at stat.math.ethz.ch
> > Subject: Re: [R] Reasons to Use R
>
> [snip]
>
> > So what's the big deal about S using files instead of memory
> > like R. I don't get the point. Isn't there enough swap space
> > for S? (Who cares
> > anyway: it works, isn't it?) Or are there any problems with S
> > and large datasets? I don't get it. You use them, Greg. So
> > you might discuss that issue.
> >
> > Wilfred
> >
> >
>
> This is my understanding of the issue (not anything official).
>
> If you use up all the memory while in R, then the OS will start swapping
> memory to disk, but the OS does not know what parts of memory correspond
> to which objects, so it is entirely possible that the chunk swapped to
> disk contains parts of different data objects, so when you need one of
> those objects again, everything needs to be swapped back in.  This is
> very inefficient.
>
> S-PLUS occasionally runs into the same problem, but since it does some
> of its own swapping to disk it can be more efficient by swapping single
> data objects (data frames, etc.).  Also, since S-PLUS is already saving
> everything to disk, it does not actually need to do a full swap, it can
> just look and see that a particular data frame has not been used for a
> while, know that it is already saved on the disk, and unload it from
> memory without having to write it to disk first.
>
> The g.data package for R has some of this functionality of keeping data
> on the disk until needed.
>
> The better approach for large data sets is to only have some of the data
> in memory at a time and to automatically read just the parts that you
> need.  So for big datasets it is recommended to have the actual data
> stored in a database and use one of the database connection packages to
> only read in the subset that you need.  The SQLiteDF package for R is
> working on automating this process for R.  There are also the bigdata
> module for S-PLUS and the biglm package for R have ways of doing some of
> the common analyses using chunks of data at a time.  This idea is not
> new.  There was a program in the late 1970s and 80s called Rummage by
> Del Scott (I guess technically it still exists, I have a copy on a 5.25"
> floppy somewhere) that used the approach of specify the model you wanted
> to fit first, then specify the data file.  Rummage would then figure out
> which sufficient statistics were needed and read the data in chunks,
> compute the sufficient statistics on the fly, and not keep more than a
> couple of lines of the data in memory at once.  Unfortunately it did not
> have much of a user interface, so when memory was cheap and datasets
> only medium sized it did not compete well, I guess it was just a bit too
> ahead of its time.
>
> Hope this helps,
>
>
>
> --
> Gregory (Greg) L. Snow Ph.D.
> Statistical Data Center
> Intermountain Healthcare
> greg.snow at intermountainmail.org
> (801) 408-8111
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
WenSui Liu
A lousy statistician who happens to know a little programming
(http://spaces.msn.com/statcompute/blog)