[Rd] readLines() fails on non-blocking connections when encoding="UTF-8" or encoding="ASCII"

[Peter Meilstrup]

Hello R-devel,

I have been trying to wrap my head around non-blocking connections and
have not been having them behave as advertised.

The issue I am having is that readLines() gets "stuck." If it reaches
the end of a stream once, it does not ever return any more data, even
when more is available on the stream. It turns out this behavior
happens when I specify encoding="ASCII" or encoding="UTF-8". However
things behave more as expected if I use encoding="native.enc".

The following code demonstrates what is happening, using socket
connections. I also observe this behavior for fifo() and
file(blocking=FALSE) connections:

sock <- serverSocket(45678)
# encoding="native.enc" works; encoding="UTF-8" fails; encoding="ASCII" fails
outgoing <- socketConnection("localhost", 45678, encoding="UTF-8")
incoming <- socketAccept(sock, encoding="UTF-8")
writeLines("hello", outgoing)
flush(outgoing)
readLines(incoming, 1) # "hello"
readLines(incoming, 1) # character(0)
writeLines("again", outgoing)
flush(outgoing)
socketSelect(list(incoming)) #TRUE
readLines(incoming, 1) # I get character(0) (incorrect)
socketSelect(list(incoming), timeout=0) #TRUE, so there is still data?
writeLines("again", outgoing)
writeLines("again", outgoing)
flush(outgoing)
readLines(incoming, 1) # character(0)
isIncomplete(incoming) # FALSE, which also seems wrong bc there is unread data?
readChar(incoming, 100) # "again\nagain\nagain\n", so readChar saw
what readLines() did not
close(incoming)
close(outgoing)
close(sock)

I have observed this with recent versions of R installed on Debian and
OSX. I also found a report of similar behavior from 2018:

https://stat.ethz.ch/pipermail/r-devel/2018-April/075883.html

Note that that report did not mention encodings being an issue;
perhaps the original bug was fixed for native encoding only?

Peter Meilstrup