[Rd] An interesting chat with ChatGPT
Boris Steipe
bor|@@@te|pe @end|ng |rom utoronto@c@
Mon Feb 13 16:54:43 CET 2023
Duncan -
Indeed, this has now been well documented; I have called these constructions "Schrödinger Facts", since they arise from a superposition of truths in the training data that collapse into an untruth when observed.
https://sentientsyllabus.substack.com/p/chatgpts-achilles-heel
Now, the curious question is: why can it program. Why is its production of R-syntax less vulnerable to this effect than a literature quote, or reference? Maybe it has to do with the fact that in producing correct syntax there is a ground truth - errors just will not run. But I'm not sure.
Regardless: it signals that we need a reinvigorated culture of validation. Actually ChatGPT will happily make test data for you. Whisvh. in a way, it judt did ;-)
Boris
> On 2023-02-13, at 08:37, Duncan Murdoch <murdoch.duncan using gmail.com> wrote:
>
> I was wondering which anchor would match in the regular expression "(a)|(b)" if both "a" and "b" matches the target string. What appears to happen is that the first match wins, and that's how it is documented in Python, but I was looking for docs on this in R.
>
> As far as I can see, the ?regex help page doesn't resolve this. So just for fun, I decided to ask ChatGPT, and got an interesting response. I won't show the whole conversation here, but just some extracts:
>
> ---------------------------------------
> Me: In a regular expression in R, if both "a" and "b" match, which does "(a)|(b)" match? Please include authoritative references in your answer.
>
> ChatGPT: [long answer about grep]
>
> Me: In grep(), it doesn't matter which of a or b matches. It's only in sub() or gsub() that it makes a difference. Do you know an authoritative reference that discusses those functions?
>
> ChatGPT: [long irrelevant answer about sub and gsub, with a reference to "R Cookbook" by P. Teetor.]
>
> Me: Teetor's book is not authoritative, and the other reference doesn't discuss ambiguous alternation.
>
> ... finally we get to the interesting part ...
>
> ChatGPT: I apologize for the mistake in my previous answer.
>
> A more authoritative reference for the behavior of regular expressions in R can be found in the official R documentation.
>
> Regarding the behavior of ambiguous alternation, the regex section of the manual "An Introduction to R" explains it as follows:
>
> "When several alternatives are possible, the first alternative is chosen, so it is important to specify alternatives in the order in which they should be tried."
>
> Here is the reference for the official R documentation:
>
> "An Introduction to R" manual: https://cran.r-project.org/doc/manuals/r-release/R-intro.html#Regular-expression-facilities
>
> I hope this reference provides the information you were looking for.
> ---------------------------------------
>
>
> All right, an authoritative answer! The only problem is that *it doesn't exist*. There's no such anchor in that document. R-intro isn't a very rapidly changing document, so I could look at every change recorded in svn, and that section has *never* existed.
>
> I guess ChatGPT is a pattern matching machine, and if such a question had an answer, it would probably be in a location like that. (I would have guessed a help page like ?regex instead of R-intro, but I had just told it that answer was wrong.)
>
> Duncan Murdoch
>
> ______________________________________________
> R-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
--
Boris Steipe MD, PhD
Professor em.
Department of Biochemistry
Temerty Faculty of Medicine
University of Toronto
More information about the R-devel
mailing list