[Rd] R 2.9.2 crashes when sorting latin1-encoded strings

Simon Urbanek simon.urbanek at r-project.org
Wed Sep 30 16:55:04 CEST 2009 

Stefan,

On Sep 30, 2009, at 5:11 , Stefan Evert wrote:

> Hi everyone!
>
> I think I stumbled over a bug in the latest R 2.9.2 patched for OS X:
>
>> R version 2.9.2 Patched (2009-09-24 r49861)
>> i386-apple-darwin9.8.0
>
>
> When I try to sort latin1-encoded character vectors, R sometimes  
> crashes with a segmentation fault.  I'm running OS X 10.5.8 and have  
> observed this behaviour both with the i386 and x86_64 builds, in the  
> R.app GUI as well as on the command line.
>
> Here's a minimal example that reliably triggers the crash on my  
> machine:
>
> =====
> print(sessionInfo())
>
> words <- c("aa", "ab", "a\xfc", "a\xe4", "b\xe4", "b\xfc", "\xe4\xfc")
> str(words)
>
> print(table(Encoding(words)))
> Encoding(words) <- "latin1"  # this is the correct encoding!
> print(table(Encoding(words)))
>
> N <- 1000
> words <- rep(words, length.out=N)
>
> print(N)
> for (i in 1:N) {
> x <- words[1:i]
> # the following line will crash for some i, depending on the  
> particular
> # strings in <words> and the subset selected for <x> above
> order(x)
> }
> =====
>
> The output I get from this code is appended at the end of the mail.  
> Note that R incorrectly declares the latin1 strings in <word> to  
> have UTF-8 encoding (this seems wrong to me because the \x escapes  
> insert raw bytes into the string).

It is correct, because you're in a UTF-8 locale (see l10n_info()) so  
all strings are UTF-8 by default - you're just manually creating a  
string that is not valid in UTF-8.


> The crash only occurs if the correct "latin1" encoding (or  
> "unknown") is explicitly specified. Otherwise the string handling  
> code appears to ignore everything after the first invalid multibyte  
> character.
>
> I haven't been able to trigger the bug without some kind of loop.   
> The crash always occurs at the same iteration, but this changes  
> depending on the contents of <words> and the specific subset  
> selected in each loop iteration.  Also note that the 64-bit version  
> of R gives a different error message.  If I omit the unrelated  
> statement "print(N)", the 64-bit version segfaults and the 32-bit  
> version just hangs with high CPU load. All this suggests to me that  
> there must be some insidious memory corruption or stack/range  
> overflow in the internal ordering code.
>

Yup:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: 13 at address: 0x0000000000000000
0x0000000100167e0d in R_gc_internal (size_needed=1) at ../../../../ 
R-2.9-branch/src/main/memory.c:1327
1327	    PROCESS_NODES();
(gdb) bt
#0  0x0000000100167e0d in R_gc_internal (size_needed=1) at ../../../../ 
R-2.9-branch/src/main/memory.c:1327
#1  0x000000010016a2bf in Rf_allocVector (type=607, length=0)  
at ../../../../R-2.9-branch/src/main/memory.c:1991
#2  0x000000010016aa65 in R_alloc (nelem=<value temporarily  
unavailable, due to optimizations>, eltsize=<value temporarily  
unavailable, due to optimizations>) at ../../../../R-2.9-branch/src/ 
main/memory.c:1669
#3  0x000000010020f316 in Rf_translateCharUTF8 (x=<value temporarily  
unavailable, due to optimizations>) at ../../../../R-2.9-branch/src/ 
main/sysutils.c:858
#4  0x0000000100216140 in Rf_Scollate (a=0x1023c1518, b=0x0)  
at ../../../../R-2.9-branch/src/main/util.c:1691
#5  0x00000001001f894e in orderVector1 (indx=<value temporarily  
unavailable, due to optimizations>, n=<value temporarily unavailable,  
due to optimizations>, key=0x11b024c00, nalast=TRUE, decreasing=FALSE,  
rho=0x1020a4778) at ../../../../R-2.9-branch/src/main/sort.c:846
#6  0x00000001001f9605 in orderVector [inlined] () at ../../../../ 
R-2.9-branch/src/main/sort.c:888
#7  do_order (call=<value temporarily unavailable, due to  
optimizations>, op=<value temporarily unavailable, due to  
optimizations>, args=0x11843fc38, rho=<value temporarily unavailable,  
due to optimizations>) at ../../../../R-2.9-branch/src/main/sort.c:891

Note that b=0x0 in the call to Rf_Scollate -- seems like some array  
overflow in the sorting code... will need some more investigation ...


In the meantime I can offer you a work-around -- working with non- 
native strings (latin1 in your case) is very expensive because they  
get converted all the time into the native locale, so you want to run
words<-iconv(words,"latin1","")

and then proceed - it's faster and doesn't crash ;).


> Can other people reproduce this problem on different platforms and  
> possibly with different versions of R?
>
>
> BTW, I ran into the crash when trying to read.delim() a file in  
> latin1 encoding, using either encoding="latin1" or  
> fileEncoding="latin1", and then converting it back and forth between  
> a character vector and a factor.  I still don't understand what's  
> going on there.  The behaviour of read.delim() seems to depend very  
> much on my locale settings when running R, which is rather unpleasant.

?? The whole point of a locale is that it declares how you are going  
to interact with the system. Handling of strings is entirely different  
depending on the encoding used by the locale - and that is the point  
of locales. When you are dealing with text (e.g. as files) you must  
always take the encoding into account and by default they are assumed  
to be in the same encoding as your locale - you really wouldn't want R  
to suddenly read all files as let's say eucJP even though your locale  
is UTF-8 ...


>  Is there a way to find out how strings are stored internally (i.e.  
> getting the exact byte representation) and whether R believes them  
> to be in UTF-8 or latin1 encoding?
>

charToRaw() will show you the raw bytes and you define using  
Encoding() how you want the string to be interpreted (supported is  
UTF-8, latin1 and unknown). If the encoding is known, R will convert  
it where needed. Normally R uses the native encoding of the locale  
you're running in. If you are dealing with files from other locales,  
you have to tell R accordingly - in most cases it's better to re- 
encode the strings (?iconv) than to work with the foreign encoding.

Cheers,
Simon



>
>
> Output of sample code on my machine:
>
>> > print(sessionInfo())
>> R version 2.9.2 Patched (2009-09-24 r49861)
>> i386-apple-darwin9.8.0
>>
>> locale:
>> en_GB/en_GB/C/C/en_GB/en_GB
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>> >
>> > words <- c("aa", "ab", "a\xfc", "a\xe4", "b\xe4", "b\xfc",  
>> "\xe4\xfc")
>> > str(words)
>> chr [1:7] "aa" "ab" "a\xfc" "a\xe4" "b\xe4" "b\xfc" ...
>> > print(table(Encoding(words)))
>>
>> unknown   UTF-8
>>    2       5
>> >
>> > Encoding(words) <- "latin1"  # this is the correct encoding!
>> > print(table(Encoding(words)))
>>
>> latin1 unknown
>>    5       2
>> >
>> > N <- 1000
>> > words <- rep(words, length.out=N)
>> >
>> > print(N)
>> [1] 1000
>> > for (i in 1:N) {
>> +   x <- words[1:i]
>> +   # the following line will crash for some i, depending on the  
>> particular
>> +   # strings in <words> and the subset selected for <x> above
>> +   order(x)
>> + }
>>
>> *** caught bus error ***
>> address 0x86, cause 'non-existent physical address'
>>
>> Traceback:
>> 1: order(x)
>> aborting ...
>> Bus error
>
> 64-bit version:
>
>> > print(sessionInfo())
>> R version 2.9.2 Patched (2009-09-24 r49861)
>> x86_64-apple-darwin9.8.0
>>
>> locale:
>> en_GB/en_GB/C/C/en_GB/en_GB
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>> >
>> > words <- c("aa", "ab", "a\xfc", "a\xe4", "b\xe4", "b\xfc",  
>> "\xe4\xfc")
>> > str(words)
>> chr [1:7] "aa" "ab" "a\xfc" "a\xe4" "b\xe4" "b\xfc" ...
>> > print(table(Encoding(words)))
>>
>> unknown   UTF-8
>>    2       5
>> >
>> > Encoding(words) <- "latin1"  # this is the correct encoding!
>> > print(table(Encoding(words)))
>>
>> latin1 unknown
>>    5       2
>> >
>> > N <- 1000
>> > words <- rep(words, length.out=N)
>> >
>> > print(N)
>> [1] 1000
>> > for (i in 1:N) {
>> +   x <- words[1:i]
>> +   # the following line will crash for some i, depending on the  
>> particular
>> +   # strings in <words> and the subset selected for <x> above
>> +   order(x)
>> + }
>> Error in order(x) : 'translateCharUTF8' must be called on a CHARSXP
>> Execution halted
>>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>

More information about the R-devel mailing list