[Rd] Error in substring: invalid multibyte string

[Toby Hocking]

Thanks for the quick response Ivan. readLines with encoding='latin1' works
for me (on Ubuntu).

However I was more concerned with the inconsistency in results between
substr and regexpr. I was expecting that if one of them errors because of
an unknown encoding then the other should as well. Even better, if regexpr
works, why shouldn't substr work as well?

Incidentally the analogous stringi function stri_sub works fine in this
case:

> stringi::stri_sub("<I>Jens Oehlschl\xe4gel-Akiyoshi", 1, 100)
[1] "<I>Jens Oehlschl\xe4gel-Akiyoshi"

But the stringi analog to nchar gives a similar warning:

> stringi::stri_length("<I>Jens Oehlschl\xe4gel-Akiyoshi")
[1] NA
Warning message:
In stringi::stri_length("<I>Jens Oehlschl\xe4gel-Akiyoshi") :
  invalid UTF-8 byte sequence detected; try calling stri_enc_toutf8()


On Sat, Jun 27, 2020 at 2:12 AM Ivan Krylov <krylov.r00t using gmail.com> wrote:

> On Fri, 26 Jun 2020 15:57:06 -0700
> Toby Hocking <tdhock5 using gmail.com> wrote:
>
> >invalid multibyte string at '<e4>gel-A<6b>iyoshi'
>
> >https://stat.ethz.ch/pipermail/r-devel/1999-November/author.html
>
> The server says that the text is UTF-8:
>
> curl -sI \
>  https://stat.ethz.ch/pipermail/r-devel/1999-November/author.html | \
>  grep Content-Type
> # Content-Type: text/html; charset=UTF-8
>
> But it's not, at least not all of it. If you ask readLines to mark
> the text as Latin-1, you get Jens Oehlschlägel-Akiyoshi without the
> mojibake and invalid multi-byte characters:
>
> x <- readLines(
>  'https://stat.ethz.ch/pipermail/r-devel/1999-November/author.html',
>  encoding = 'latin1'
> )[28]
> substr(x, 1, 100)
> # [1] "<I>Jens Oehlschlägel-Akiyoshi"
>
> The behaviour we observe when encoding = 'latin1' is not specified
> results from returned lines having "unknown" encoding. The substr()
> implementation tries to interpret such strings according to multi-byte C
> locale rules (using mbrtowc(3)). On my system (yours too, probably, if
> it's GNU/Linux or macOS), the multi-byte C locale encoding is UTF-8,
> and this Latin-1 string does not result in valid code points when
> decoded as UTF-8.
>
> --
> Best regards,
> Ivan
>

	[[alternative HTML version deleted]]