[R] Request for advice on character set conversions (those damn Excel files, again ...)

[Peter Dalgaard]

Emmanuel Charpentier wrote:
> On Mon, 08 Sep 2008 01:45:51 +0200, Peter Dalgaard wrote :
>
>   
>> Emmanuel Charpentier wrote:
>>     
>>> Dear list,
>>>         
>>>       
>
> [ Snip ... ]
>
>   
>> This looks reasonably sane, I think. The last loop could be d[] <-
>> lapply(d, conv1, from, to), but I think that is cosmetic. You can't
>> really do much better because there is no simple way of distinguishing
>> between the various 8-bit character sets.
>>     
>
> Thank you Peter !
>
> Could you point me to some not-so-simple (or even doubleplusunsimple) 
> ways ? I get the problem not so rarely, and I'd like to pull this chard 
> outta my poor tired foot one and for all... and I suppose that I am not 
> alone in this predicament.
>
>   
In full generality it is impossible, but you might get something if you 
make certain assumptions. If you can convert from UTF8, then it probably 
is UTF8 (or ASCII but in either case, you're done). Otherwise it is a 
single-byte 8-bit encoding if  the language can be assumed to be 
French.  If it uses characters between 0x80 and 0x9f, then it is not 
latin1 but rather cp437, 850, or 1252.   The tricky bit is that although 
the presence of say 0x82 suggests that it is not cp1252, but rather 437 
or 850 (e aigu) , it just might be 1252 after all (single low quote). 
Some sort of naive Bayes classifier might work.

>>                                           You could presumably setup
>> some heuristics. like the fact that the occurrence of 0x82 or 0x8a
>> probably indicates cp437, but it gets tricky. (At least, in French, you
>> don't have the Danish/Norwegian peculiarity that upper/lowercase o-slash
>> were missing in cp437, and therefore often replaced yen and cent symbols
>> in matrix printer ROMs. We still get the occational parcel addressed to
>> "¥ster Farimagsgade".)
>>     
>
> Peter, you're gravely underestimating the ingenuity of some Excel 
> l^Husers... (and your story is a possible candidate for a fortune() 
> entry...)

(If so, please respell "occasional". Ouch! And actually, the cent/yen 
thing is also a difference between cp437 and cp850, so the story may be 
a bit too colourful.)

-- 
   O__  ---- Peter Dalgaard             Øster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark      Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)              FAX: (+45) 35327907