[R] Handling large SAS file in R

[Milan Bouchet-Valat]

Le mardi 28 janvier 2014 à 11:51 -0800, Fisher Dennis a écrit :
> Marc
> 
> I understand that R is staffed entirely by volunteers and I appreciate
> the massive efforts that have created this awesome software. My
> suggestion was aiming to encourage the Core team that one weakness of
> the present structure is less-than-optimal interchangeability with
> SAS.  If this were improved, it would be a great service to the R
> community, particularly those of us in the pharmaceutical development
> world who must interface with SAS.
> 
> I just wrote to David Smith at Revolution Analytics, encouraging him
> to consider taking this on (as you suggested).
FWIW, a new pure-C library to read SAS, SPSS and Stata files has been
released here:
https://github.com/WizardMac/ReadStat

I'm not sure whether it constitutes an improvement over the sas7bdat
package in terms of features, but in terms of speed it should perform
much better since it's written in C. With a reasonable amount of work it
could probably be adapted to work in R.


My two cents

> Dennis
> 
> Dennis Fisher MD
> P < (The "P Less Than" Company)
> Phone: 1-866-PLessThan (1-866-753-7784)
> Fax: 1-866-PLessThan (1-866-753-7784)
> www.PLessThan.com
> 
> 
> 
> On Jan 28, 2014, at 8:43 AM, Marc Schwartz <marc_schwartz at me.com> wrote:
> 
> > Dennis,
> > 
> > The key difference is that with R, you are, as always, dependent upon volunteers providing software at no charge to you, most of whom have full time (and then some) jobs. Those jobs (and in many cases, family) will be their priority, as I am sure is the case with Matt. 
> > 
> > Unless they are in a position where their employer specifically allows them to allocate a percentage of their work time to voluntary projects, like R, you are at the inevitable mercy of that volunteer's time and priorities.
> > 
> > In the case of Stat/Transfer, they are a profit motivated business with revenue tied directly to the sales of the application. Thus, they have a very different perspective on serving their paying customers and can allocate dedicated resources to the functionality in their application.
> > 
> > An alternative here would be for one of the for profit companies that sell and support R versions, to take on the task of providing some of these facilities and providing them back to the community as a service. But, that is up to them to consider in their overall business plan and the value that they perceive it brings to their products.
> > 
> > Regards,
> > 
> > Marc Schwartz
> > 
> > On Jan 28, 2014, at 9:59 AM, Dennis Fisher <fisher at plessthan.com> wrote:
> > 
> >> Colleagues
> >> 
> >> Frank Harrell wrote that “you need to purchase Stat/Transfer", which I did many years ago and continue to use.  
> >> 
> >> But I don’t understand why the sas7bdat package (or something equivalent) cannot reverse engineer the SAS procedures so that R users can read sas7bdat files as well as StatTransfer.  I have been in contact with the maintainer, Matt Shotwell, regarding bugs in the present version (0.4) and he wrote:
> >> 	it tends to languish just one or two items from the top of my TODO... I hope to get back to it soon.
> >> I have also written to this bulletin board about the foreign package not being able to process certain SAS XPT files (which StatTransfer handled without any problem).
> >> 
> >> I am a strong advocate of R and I have arranged work-arounds (using StatTransfer) in these cases.  However, R users would benefit from the ability of R to read any SAS file without intermediate software.   I would offer to participate in any efforts to accomplish this but I think that it is beyond my capabilities.  
> >> 
> >> Dennis
> >> 
> >> Message: 23
> >> Date: Mon, 27 Jan 2014 13:25:54 -0800 (PST)
> >> From: Frank Harrell <f.harrell at vanderbilt.edu>
> >> To: r-help at r-project.org
> >> Subject: Re: [R] Handlig large SAS file in R
> >> Message-ID: <1390857954542-4684250.post at n4.nabble.com>
> >> Content-Type: text/plain; charset=us-ascii
> >> 
> >> For that you need to purchase Stat/Transfer.
> >> Frank
> >> 
> >> 
> >> hans012 wrote
> >>> Hey Guys
> >>> I have a .sas7bdat file of 1.79gb that i want to read.
> >>> I am using the .sas7bdat package to read the file and after i typed the
> >>> command read.sas7bdat('filename.sas7bdat') it has been 3 hours with no
> >>> result so far.
> >>> Is there a way that i can see the progress of the read? 
> >>> Or is there another way to read the file with less computing time?
> >>> I do not have access to SAS, the file was sent to me.
> >>> 
> >>> Let me know what you guys think
> >>> KR
> >>> Hans
> >> 
> >> 
> >> Dennis Fisher MD
> >> P < (The "P Less Than" Company)
> >> Phone: 1-866-PLessThan (1-866-753-7784)
> >> Fax: 1-866-PLessThan (1-866-753-7784)
> >> www.PLessThan.com
> > 
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.