[Rd] NEWS item for bugfix in normalizePath and file.exists?

[Toby Hocking]

Hi Tomas, thanks for the thoughtful reply. That makes sense about the
problems with C locale on windows. Actually I did not choose to use C
locale, but instead it was invoked automatically during a package check.
To be clear, I do NOT have a file with that name, but I do want file.exists
to return a reasonable value, FALSE (with no error). If that behavior is
unspecified, then should I use something like tryCatch(file.exists(x),
error=function(e)FALSE) instead of assuming that file.exists will always
return a logical vector without error? For my particular application that
work-around should probably be sufficient, but one may imagine a situation
where you want to do

x <- "\360\237\247\222\n| \360\237\247\222\360\237\217\273\n|
\360\237\247\222\360\237\217\274\n| \360\237\247\222\360\237\217\275\n|
\360\237\247\222\360\237\217\276\n| \360\237\247\222\360\237\217\277\n"
Encoding(x) <- "unknown"
Sys.setlocale(locale="C")
f <- tempfile()
cat("", file = f)
two <- c(x, f)
file.exists(two)

and in that case the correct response from R, in my opinion, would be
c(FALSE, TRUE) -- not an error.
Toby

On Wed, Apr 28, 2021 at 3:10 AM Tomas Kalibera <tomas.kalibera using gmail.com>
wrote:

> Hi Toby,
>
> a defensive, portable approach would be to use only file names regarded
> portable by POSIX, so characters including ASCII letters, digits,
> underscore, dot, hyphen (but hyphen should not be the first character).
> That would always work on all systems and this is what I would use.
>
> Individual operating systems and file systems and their configurations
> differ in which additional characters they support and how. On some,
> file names are just sequences of bytes, on some, they have to be valid
> strings in certain encoding (and then with certain exceptions).
>
> On Windows, file names are at the lowest level in UTF-16LE encoding (and
> admitting unpaired surrogates for historical reasons). R stores strings
> in other encodings (UTF-8, native, Latin-1), so file names have to be
> translated to/from UTF-16LE, either directly by R or by Windows.
>
> But, there is no way to convert (non-ASCII) strings in "C" encoding to
> UTF16-LE, so the examples cannot be made to work on Windows.
>
> When the translation is left on Windows, it assumes the non-UTF-16LE
> strings are in the Active Code Page encoding (shown as "system encoding"
> in sessionInfo() in R, Latin-1 in your example) instead of the current C
> library encoding ("C" in your example). So, file names coming from
> Windows will be either the bytes of their UTF-16LE representation or the
> bytes of their Latin-1 representation, but which one is subject to the
> implementation details, so the result is really unusable.
>
> I would say using "C" as encoding in R is not a good idea, and
> particularly not on Windows.
>
> I would say that what happens with such file names in "C" encoding is
> unspecified behavior, which is subject to change at any time without
> notice, and that both the R 4.0.5 and R-devel behavior you are observing
> are acceptable. I don't think it should be mentioned in the NEWS.
> Personally, I would prefer some stricter checks of strings validity and
> perhaps disallowing the "C" encoding in R, so yet another behavior where
> it would be clearer that this cannot really work, but that would require
> more thought and effort.
>
> Best
> Tomas
>
>
> On 4/27/21 9:53 PM, Toby Hocking wrote:
>
> > Hi all, Today I noticed bug(s?) in R-4.0.5, which seem to be fixed in
> > R-devel already. I checked on
> > https://developer.r-project.org/blosxom.cgi/R-devel/NEWS and there is no
> > mention of these changes, so I'm wondering if they are intentional? If
> so,
> > could someone please add a mention of the bugfix in the NEWS?
> >
> > The problem involves file.exists, on windows, when a long/strange input
> > file name Encoding is unknown, in C locale. I expected that FALSE should
> be
> > returned (and it is on R-devel), but I got an error in R-4.0.5. Code to
> > reproduce is:
> >
> > x <- "\360\237\247\222\n| \360\237\247\222\360\237\217\273\n|
> > \360\237\247\222\360\237\217\274\n| \360\237\247\222\360\237\217\275\n|
> > \360\237\247\222\360\237\217\276\n| \360\237\247\222\360\237\217\277\n"
> > Encoding(x) <- "unknown"
> > Sys.setlocale(locale="C")
> > sessionInfo()
> > file.exists(x)
> >
> > Output I got from R-4.0.5 was
> >
> >> sessionInfo()
> > R version 4.0.5 (2021-03-31)
> > Platform: x86_64-w64-mingw32/x64 (64-bit)
> > Running under: Windows 10 x64 (build 19042)
> >
> > Matrix products: default
> >
> > locale:
> > [1] C
> > system code page: 1252
> >
> > attached base packages:
> > [1] stats     graphics  grDevices utils     datasets  methods   base
> >
> > loaded via a namespace (and not attached):
> > [1] compiler_4.0.5
> >> file.exists(x)
> > Error in file.exists(x) : file name conversion problem -- name too long?
> > Execution halted
> >
> > Output I got from R-devel was
> >
> >> sessionInfo()
> > R Under development (unstable) (2021-04-26 r80229)
> > Platform: x86_64-w64-mingw32/x64 (64-bit)
> > Running under: Windows 10 x64 (build 19042)
> >
> > Matrix products: default
> >
> > locale:
> > [1] C
> >
> > attached base packages:
> > [1] stats     graphics  grDevices utils     datasets  methods   base
> >
> > loaded via a namespace (and not attached):
> > [1] compiler_4.2.0
> >> file.exists(x)
> > [1] FALSE
> >
> > I also observed similar results when using normalizePath instead of
> > file.exists (error in R-4.0.5, no error in R-devel).
> >
> >> normalizePath(x) #R-4.0.5
> > Error in path.expand(path) : unable to translate 'p'
> > | p'p;
> > | p'p<
> > | p'p=
> > | p'p>
> > | p'p<bf>
> > ' to UTF-8
> > Calls: normalizePath -> path.expand
> > Execution halted
> >
> >> normalizePath(x) #R-devel
> > [1] "C:\\Users\\th798\\R\\\360\237\247\222\n|
> > \360\237\247\222\360\237\217\273\n| \360\237\247\222\360\237\217\274\n|
> > \360\237\247\222\360\237\217\275\n| \360\237\247\222\360\237\217\276\n|
> > \360\237\247\222\360\237\217\277\n"
> > Warning message:
> > In normalizePath(path.expand(path), winslash, mustWork) : path[1]="🧒
> > | 🧒🏻
> > | 🧒🏼
> > | 🧒🏽
> > | 🧒🏾
> > | 🧒🏿
> > ": The filename, directory name, or volume label syntax is incorrect
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-devel using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>

	[[alternative HTML version deleted]]