[R-sig-Geo] Re: About dealing 2-byte code characters(ex. Japanese) in new maptools

[Roger Bivand]

Dear Hisaji,

A couple of questions before we include Prof. Ripley, who put the 
functions into the foreign package:

Platform: OS version and R version? Did you install from source both 
times? If from source is the build train the same?

Can you help by trying to see what happens if you remove/install maptools, 
looking at http://cran.r-project.org/src/contrib/Archive, install 0.3-8, 
then step through to see whether the change in behaviour happened at the 
0.4-7/0.4-8 border, or maybe earlier (in which case I think it will be me, 
not him who did something). I'm afraid for Windows XP that you need to 
have the full package build train installed, which maybe you have anyway:

http://www.murdoch-sutherland.com/Rtools/

If something has changed in that, 0.3-8 would now give the non-2-byte 
characters.

I can understand that this is important, so I'd like to find out why - my 
first guess would be a change in your operating environment, but if a 
newly installed 0.3-8 on R 2.0.0 gives 2-byte characters on your system, 
then we can work forward and see where the change happened. I did make 
changes in reading character strings about 0.4-5 (Nicholas has hardcoded a 
maximum string length to 50 bytes, and a user file had a very long > 200 
single-byte character string value, so I checked the lengths of the 
strings first and allocated). This is in Rdbfread.c, traces at line about 
152 and about 195 - look in versions 0.4-4 and 0.4-5 to see the 
difference. If 0.4-4 still reads 2-byte, and 0.4-5 doesn't, that's it, and 
we can try to find out how to get the equivalent file in foreign to do the 
same.

Sorry for this inconvenience, but we'll need your help to find out what is 
going on here.

Best wishes,

Roger

On Fri, 29 Oct 2004, Hisaji ONO wrote:

> Hello, Professor Bivand.
> 
>  Thank you for updating maptools.
> 
>  I've tried to use this.
> 
>  And I've noticed this couldn't deal with Japanese
> Characters, which were included dbase files.
> 
> 
>  Former version(0.3-8),
> 
> > raildb<-maptools:::dbf.read("/railway.dbf")
> > raildb
>                                         NAME
> 1                               $B#J#R2#?\2l@~(B
> 2                               $B#J#R2#?\2l@~(B
> 
>  In Japanese WinXP environment, these characters were
> displayed correctly.
> 
> 
>  However latest version(0.4-8),
> 
> 
> > raildb<-read.dbf("/railway.dbf",as.is=TRUE)
> 
> > raildb$NAME[1:2]
> [1] "&#8218;i&#8218;q$B"s!#(B&#144;{$B"s(B&ecirc;&#144;&uuml;"    
>    
> [2] "&#8218;i&#8218;q$B"s!#(B&#144;{$B"s(B&ecirc;&#144;&uuml;" 
> 
>  These aren't Japanese.
> 
>  How come?
> 
> 
>  Regards.
> 
> 
> 

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand at nhh.no