[Rd] Problem with read.xport() from foreigh package (PR#7389)

[ripley at stats.ox.ac.uk]

The relevant part of the code is

 		    if(strlen(tmpchar) == 1 && IS_SASNA_CHAR(tmpchar[0]))

#define IS_SASNA_CHAR(c) ((c) == 0x5f || (c) == 0x2e || \
                           (0x41 <= (c) && (c) <= 0x5a))

which says that single-character fields containing ., _, A-Z are to be 
taken as missing values.  That is true of all single-character fields in 
this file.

Looking at the reference, it says

   Missing values are written out with the first byte (the exponent)
   indicating the proper missing values. All subsequent bytes are 0x00. The
   first byte is:

      type      byte
       ._       0x5f
       .        0x2e
       .A       0x41
       .B       0x42
          ....
       .Z       0x5a

which suggests this is intended to apply only to numeric records 
('exponent'), whereas R applies it only to character records.  Elsewhere I 
found

   SAS stores 'missing' data depending on the variables data type:
   character or numeric. A 'missing' character value is represented as a
   blank (' ') or null (''). A 'missing' numeric value is represented as a
   single dot (.). SAS allows you to differentiate between values that are
   missing for different reasons. For example, in survey research a
   question may not be answered because the respondent refuses to answer or
   because they don't know the answer. SAS has a range of missing values to
   cover this case. These special missing values are for numerics only and
   are the letters of the alphabet preceded by the single dot .A thru .Z

which seems to confirm this.

Does anyone know for sure?


On Wed, 24 Nov 2004, Ruskin Chow wrote:

> Dear Prof. Ripley,
>
> Thanks for your prompt reply. In fact when I converted one-char data fields
> in SAS to two-char fields before generating the SAS transport format file,
> the problem (that all single char fields become <NA>) disappears. The
> problem is readily reproducible and I attach two sample data files in SAS
> transport format, one with one-char data fields ("char1.xport") and the
> other data file ("char2.xport") with the same data fields except that
> one-char fields are converted to two-char fields (for example, see the
> fields named "SEX", "GB" etc.). The R output text file ("lastsave.txt") is
> also attached so that you can verify it easily at your end.
>
> Even with the above, there is no conclusive evidence whether this problem is
> actually (a) with SAS in generating the transport file, or (b) with R in the
> read.xport() function. However, with the indication in the following
> discussion thread:
>
> http://maths.newcastle.edu.au/~rking/R/help/02a/3258.html
>
> I tend to believe that R may have the answer.
>
> Hope this helps and thanks again for your quick response.
>
> Best regards,
>
> Ruskin
>
> -----Original Message-----
> From: Prof Brian Ripley [mailto:ripley at stats.ox.ac.uk]
> Sent: Tuesday, November 23, 2004 7:02 PM
> To: ruskin at iba.com.hk
> Cc: R-bugs at biostat.ku.dk
> Subject: Re: [Rd] Problem with read.xport() from foreigh package (PR#7389)
>
> On Tue, 23 Nov 2004 ruskin at iba.com.hk wrote:
>
>> Full_Name: Ruskin Chow
>> Version: R 2.0.1
>> OS: Windows 2000
>> Submission from: (NULL) (203.169.154.66)
>>
>>
>> Data imported from SAS using read.xport() in package foreign are converted
> to
>> <NA> when the SAS data field consists of character strings that are only
> one
>> character long.
>>
>> This is apparently a previously reported bug and perhaps fixed in some
> platform
>> other than Windows (rw2001).Some discussion of the bug can be found from
> the
>> following website:
>>
>> https://stat.ethz.ch/pipermail/r-help/2002-April/019349.html
>
> There is no Windows-specific version of read.xport(), so that change
> happened everywhere: but that message does not say it was fixed nor which
> change might have fixed it.  I am guessing the change is this one
>
> 2002-03-27  Douglas Bates  <bates at stat.wisc.edu>
>
>         * src/SASxport.c (IS_SASNA_CHAR): Silly typo (0x4l, not 0x41)
>         caught by Peter.
>
> and that does not sound like the same thing.
>
>> I've downloaded the latest foreign package (foreign_0.8-1.zip) from CRAN
> but it
>> doesn't seem to work.
>   ^^^^^^^^^^^^^^^^^^^^
>
> What does that mean?  It `works' on its test suite: none of the changes
> since 0.8 are related to this question.  I don't believe there is an
> example in the test suite with one-char strings, and we need an example to
> reproduce what you are seeing.
>
> So, please read the posting guide and provide us with a reproducible
> example.
>
> --
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595