[R] [OT] A free (as in freedom) replacement for StatTransfer

[Frank E Harrell Jr]

Marc Schwartz wrote:
> On Sun, 2007-12-09 at 12:26 +1100, Tim Churches wrote:
>> Marc Schwartz wrote:
>>> Perhaps the most notable format that is lacking is the SAS proprietary
>>> format (not the Transport format), which is not openly published and to
>>> my knowledge, has not been independently reverse engineered.
>> The SAS proprietary dataset and format catalogue structures were
>> successfully reverse engineered by a small software firm called
>> Conceptual and were made available in a product called DBMS/Copy.
>> DBMS/Copy is/was similar to StatTransfer, but by 2002 was going a lot
>> further by adding support for much of the SAS data step syntax and some
>> core SAS procedures as well - in other words, it was rapidly becoming a
>> very viable and quite good pop person's SAS (with a modest one-off
>> license fee). However, the SAS Institute bought out the privately-held
>> Conceptual company, and now sells DBMS/Copy thhrough its wholly-owned
>> daat integration offshoot company, DataFlux, but without the SAS
>> datastep support features (to avoid competition with the mainstream SAS
>> cash cows) - see http://www.conceptual.com/
>>
>>> Any of the commercial products that support that format, with the
>>> possible exception of the SAS System Viewer (which is not open source,
>>> but is free from SAS), will be closed source and will have to be
>>> purchased.
>> DBMS/Copy is definitely closed-source and is probably not nearly as
>> cheapl as it once was when sold by Conceptual. But it is a great product
>> for convert to or from SAS proprietary data sets and format catalogues,
>> and works well and quickly with even huge datasets.
> 
> I am familiar with DBMS/Copy as my company uses it. Single user Windows
> licenses are not too bad (~$500 U.S.), but going beyond that gets
> expensive quickly.
> 
> I was aware of the past corporate history of Conceptual and it's
> transition to DataFlux/SAS, but not that the original developer (who's
> name escapes me at the moment) had reverse engineered the SAS format. He
> is a SAS employee now, working on other projects and the rumors
> percolate out of Tech Support in Cary every now and then that DBMS/Copy
> will be EOL'd. There have not been any updates/patches for version 8
> since January of this year.
> 
> The product is problematic however, in that it will both open and write
> SAS datasets that in actuality may not be readable by SAS itself. A
> significant problem that caused us to actually have to purchase SAS
> earlier this year to be able to validate aspects of a client data
> transfer process and we don't use SAS for anything else.
> 
> Indeed, there is an inconsistency in the behavior of SAS, DBMS/Copy and
> the SAS System Viewer when reading SAS datasets. Given that all three
> are now under the purview of SAS, it is in some respects surprising and
> in others, not so much.
> 
> Unfortunately, SAS is now bundling BASE, STAT and GRAPH as the entry
> level offering, so one cannot just get BASE as a stand-alone product any
> longer, which is all we needed. This makes the investment even more
> expensive for those of us who have to purchase at full commercial
> pricing.
> 
> Regards,
> 
> Marc

What is very telling about the true nature of the proprietary world, and 
reinforces my belief in open source software, is that the SAS Viewer 
(which works well under wine in Linux) has critical bugs in its csv or 
tab delimited data export just as PROC EXPORT under SAS-proper does.  If 
you have commas or unmatched quotes in a character field (tab for the 
tab delimeter option), the resulting ascii csv file will be broken.  To 
pay that much for SAS and to have such glaring errors that are never 
fixed is amazing.  On a side note I see that OpenOffice can read Word 
2007 documents but Word cannot read any OpenOffice documents.

I have used Stat/Transfer to great success for reading SAS binary files, 
and it runs perfectly under wine on Linux (best luck by creating Stata 
files from it and using the stata.get function in Hmisc (which uses the 
foreign package) to read the result).  I don't have to use Stat/Transfer 
very often but prefer it to DBMS/Copy as it allows more independence 
from the questionable policies of SAS Institute.
-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University