[Rd] writeLines argument useBytes = TRUE still making conversions

[Kevin Ushey]

Of course, right after writing this e-mail I tested on my Windows
machine and did not see what I expected:

> charToRaw(before)
[1] c3 a9
> charToRaw(after)
[1] e9

so obviously I'm misunderstanding something as well.

Best,
Kevin

On Sat, Feb 17, 2018 at 2:19 PM, Kevin Ushey <kevinushey at gmail.com> wrote:
> From my understanding, translation is implied in this line of ?file (from the
> Encoding section):
>
>     The encoding of the input/output stream of a connection can be specified
>     by name in the same way as it would be given to iconv: see that help page
>     for how to find out what encoding names are recognized on your platform.
>     Additionally, "" and "native.enc" both mean the ‘native’ encoding, that is
>     the internal encoding of the current locale and hence no translation is
>     done.
>
> This is also hinted at in the documentation in ?readLines for its 'encoding'
> argument, which has a different semantic meaning from the 'encoding' argument
> as used with R connections:
>
>     encoding to be assumed for input strings. It is used to mark character
>     strings as known to be in Latin-1 or UTF-8: it is not used to re-encode
>     the input. To do the latter, specify the encoding as part of the
>     connection con or via options(encoding=): see the examples.
>
> It might be useful to augment the documentation in ?file with something like:
>
>     The 'encoding' argument is used to request the translation of strings when
>     writing to a connection.
>
> and, perhaps to further drive home the point about not translating when
> encoding = "native.enc":
>
>     Note that R will not attempt translation of strings when encoding is
>     either "" or "native.enc" (the default, as per getOption("encoding")).
>     This implies that attempting to write, for example, UTF-8 encoded content
>     to a connection opened using "native.enc" will retain its original UTF-8
>     encoding -- it will not be translated.
>
> It is a bit surprising that 'native.enc' means "do not translate" rather than
> "attempt translation to the encoding associated with the current locale", but
> those are the semantics and they are not bound to change.
>
> This is the code I used to convince myself of that case:
>
>     conn <- file(tempfile(), encoding = "native.enc", open = "w+")
>
>     before <- iconv('é', to = "UTF-8")
>     cat(before, file = conn, sep = "\n")
>     after <- readLines(conn)
>
>     charToRaw(before)
>     charToRaw(after)
>
> with output:
>
>     > charToRaw(before)
>     [1] c3 a9
>     > charToRaw(after)
>     [1] c3 a9
>
> Best,
> Kevin
>
>
> On Thu, Feb 15, 2018 at 9:16 AM, Ista Zahn <istazahn at gmail.com> wrote:
>> On Thu, Feb 15, 2018 at 11:19 AM, Kevin Ushey <kevinushey at gmail.com> wrote:
>>> I suspect your UTF-8 string is being stripped of its encoding before
>>> write, and so assumed to be in the system native encoding, and then
>>> re-encoded as UTF-8 when written to the file. You can see something
>>> similar with:
>>>
>>>     > tmp <- 'é'
>>>     > tmp <- iconv(tmp, to = 'UTF-8')
>>>     > Encoding(tmp) <- "unknown"
>>>     > charToRaw(iconv(tmp, to = "UTF-8"))
>>>     [1] c3 83 c2 a9
>>>
>>> It's worth saying that:
>>>
>>>     file(..., encoding = "UTF-8")
>>>
>>> means "attempt to re-encode strings as UTF-8 when writing to this
>>> file". However, if you already know your text is UTF-8, then you
>>> likely want to avoid opening a connection that might attempt to
>>> re-encode the input. Conversely (assuming I'm understanding the
>>> documentation correctly)
>>>
>>>     file(..., encoding = "native.enc")
>>>
>>> means "assume that strings are in the native encoding, and hence
>>> translation is unnecessary". Note that it does not mean "attempt to
>>> translate strings to the native encoding".
>>
>> If all that is true I think ?file needs some attention. I've read it
>> several times now and I just don't see how it can be interpreted as
>> you've described it.
>>
>> Best,
>> Ista
>>
>>>
>>> Also note that writeLines(..., useBytes = FALSE) will explicitly
>>> translate to the current encoding before sending bytes to the
>>> requested connection. In other words, there are two locations where
>>> translation might occur in your example:
>>>
>>>    1) In the call to writeLines(),
>>>    2) When characters are passed to the connection.
>>>
>>> In your case, it sounds like translation should be suppressed at both steps.
>>>
>>> I think this is documented correctly in ?writeLines (and also the
>>> Encoding section of ?file), but the behavior may feel unfamiliar at
>>> first glance.
>>>
>>> Kevin
>>>
>>> On Wed, Feb 14, 2018 at 11:36 PM, Davor Josipovic <davorj at live.com> wrote:
>>>>
>>>> I think this behavior is inconsistent with the documentation:
>>>>
>>>>   tmp <- 'é'
>>>>   tmp <- iconv(tmp, to = 'UTF-8')
>>>>   print(Encoding(tmp))
>>>>   print(charToRaw(tmp))
>>>>   tmpfilepath <- tempfile()
>>>>   writeLines(tmp, con = file(tmpfilepath, encoding = 'UTF-8'), useBytes = TRUE)
>>>>
>>>> [1] "UTF-8"
>>>> [1] c3 a9
>>>>
>>>> Raw text as hex: c3 83 c2 a9
>>>>
>>>> If I switch to useBytes = FALSE, then the variable is written correctly as  c3 a9.
>>>>
>>>> Any thoughts? This behavior is related to this issue: https://github.com/yihui/knitr/issues/1509
>>>>
>>>>
>>>>         [[alternative HTML version deleted]]
>>>>
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel