[Rd] Importing csv files

[Frank E Harrell Jr]

Prof Brian Ripley wrote:
> I think we need to know what you mean by `large' and why read.table is 
> not fast enough (and hence if some of the planned improvements might be 
> all that is needed).

I was referring to the e-mail exchanges on r-help about read.table a few 
weeks ago, then there was a new discussion the other day concerning RAM 
usage and read.table not knowing the number of rows up front.  I believe 
that the posters provided some timings and examples.

> 
> Could you make some examples available for profiling?
> 
> It seems to me that there are some delicate licensing issues in 
> distributing a product that writes .rda format except under GPL. See, 
> for example, the GPL FAQ.

My understanding is that David is not distributing dataload any more, 
though I would not like to discourage commercial vendors (such as 
providers of Stat/Transfer and DBMSCOPY) from providing .rda output as 
an option.  I assume that new code written under GPL would not be a 
problem.  -Frank

> 
> On Thu, 23 Dec 2004, Frank E Harrell Jr wrote:
> 
>> There is a recurring need for importing large csv files quickly.  
>> David Baird's dataload is a standalone program that will directly 
>> create .rda files from .csv (it also handles many other conversions).  
>> Unfortunately dataload is no longer publicly available because of some 
>> kind of relationship with Stat/Transfer.  The idea is a good one, 
>> though.  I wonder if anyone would volunteer to replicate the csv->rda 
>> standalone functionality or to provide some Perl or Python tools for 
>> making creation of .rda files somewhat easy outside of R.
>>
>> As an aside, I routinely see 30-fold reductions in file sizes for .rda 
>> files (made with save(..., compress=TRUE)) compared with the size of 
>> SAS binary datasets.  And load( ) times are fast.
>>
>> It's been a great year for R.  Let me take this opportunity to thank 
>> the R leaders for a fantastic job that gives immeasurable benefits to 
>> the community.
> 
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University