[R] Amazing AI

Tue Dec 20 07:00:40 CET 2022

Boris,

What you are telling us is not particularly new or spectacular in a sense.
It has often been hard to grade assignments students do when they choose an
unexpected path. I had one instructor who always graded my exams (in the
multiple courses I took with him) because unlike most of the sheep, I did
not bother memorizing the way something was done or proven and created my
own solutions on the fly during the test, often in ways he had to work hard
at following and that almost amused him as mine tended to be correct, albeit
not what he would have thought of.

Your issue is not particularly about R as similar scenarios can be found in
many languages and environments.

In programming, it is arguably worse as it is common to be able to do things
so many ways. But plain old R has so many packages available, often with
source code, that any student who finds one that does something they want,
may be able to copy and modify some of the functions involved into their own
code already and fool an evaluator into thinking they did it on their own.
That is a tad harder, as many packages improve the code efficiency by
(re)writing many parts in C/C++.

I have seen things like a GUI that lets you click on various check boxes and
other controls and then use those instructions to read in data files, do
various operations on them, and provide output. Some allow quite a bit of
functionality and also offer you the opportunity to see the R code it
generates, and let you adjust that if it does not quite meet your needs.
Much of it involves including various packages and calling functions in
them, but if your students are allowed to use such things, how would you
know how little actual work they did?

I echo what someone else wrote. Training students for their future jobs in
an uncertain and changing future, may be more effective in teaching them how
to find ever better or different ways to get things done, or even switch to
growing areas near their field.  All kinds of automation of jobs are
happening and will continue to happen in the knowledge professions. People
who read manuals cover to cover and keep consulting them constantly are a
rarity. Many people often first do web searches or consult experts including
online versions of the documentation. Many will happily use software that
lets them do more and more with fewer lines of code written by them and
especially when that software has been used and tested long enough to be
relatively free of bugs when used as directed. Why would anyone these days
want to constantly re-invent the wheel and write routines to read in data
from files using known formats when you can use ones that exist and, if
needed in special cases, make some tweaks such as converting a column it
made into integer, back into the floating point you want for some later
reason?

But if your students are using something that is error-prone when used the
way they are using it, that is a problem as they are not only not learning
some basics or techniques you want them to know, but relying on what may be
bad tools without taking the time and effort to check the result or make
their own tweaks. Such software may not provide a way to do something like
treat multiple entries of various kinds as being NA, as an example. So you
would need your own code to check the result between some steps and do your
own further conversions so that "." and "" and "NA" and "-" all become NA,
again, just a made up example.

Yes, some students will easily fool you when grading but that already
happens when someone hires out getting some work done and claims it as their
own.

-----Original Message-----
From: R-help <r-help-bounces using r-project.org> On Behalf Of Boris Steipe
Sent: Monday, December 19, 2022 3:16 PM
To: Milan Glacier <news using milanglacier.com>
Cc: r-help using r-project.org
Subject: Re: [R] Amazing AI

Exactly. But not just "error prone", rather: eloquently and confidently
incorrect. And that in itself is a problem. When I evaluate students' work,
I implicitly do so from a mental model of the student - aptitude, ability,
experience, language skills etc. That's useful for summative assessment,
since it helps efficiency - but that won't work anymore. I see a need to
assess much more carefully, require fine-grained referencing, check every
single fact ... and that won't scale. And then there is also the spectre of
having to decide when this crosses the line to "concoction" - i.e. an actual
academic offence ...

Best,
Boris

> On 2022-12-19, at 03:58, Milan Glacier <news using milanglacier.com> wrote:
> 
> [You don't often get email from news using milanglacier.com. Learn why this 
> is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> On 12/18/22 19:01, Boris Steipe wrote:
>> Technically not a help question. But crucial to be aware of, especially
for those of us in academia, or otherwise teaching R. I am not aware of a
suitable alternate forum. If this does not interest you, please simply
ignore - I already know that this may be somewhat OT.
>> 
>> Thanks.
>> ------------------------------------------------------
>> 
>> You very likely have heard of ChatGPT, the conversation interface on top
of the GPT-3 large language model and that it can generate code. I thought
it doesn't do R - I was wrong. Here is a little experiment:
>> Note that the strategy is quite different (e.g using %in%, not
duplicated() ), the interpretation of "last variable" is technically correct
but not what I had in mind (ChatGPT got that right though).
>> 
>> 
>> Changing my prompts slightly resulted it going for a dplyr solution
instead, complete with %>% idioms etc ... again, syntactically correct but
not giving me the fully correct results.
>> 
>> ------------------------------------------------------
>> 
>> Bottom line: The AI's ability to translate natural language instructions
into code is astounding. Errors the AI makes are subtle and probably not
easy to fix if you don't already know what you are doing. But the way that
this can be "confidently incorrect" and plausible makes it nearly impossible
to detect unless you actually run the code (you may have noticed that when
you read the code).
>> 
>> Will our students use it? Absolutely.
>> 
>> Will they successfully cheat with it? That depends on the assignment. We
probably need to _encourage_ them to use it rather than sanction - but
require them to attribute the AI, document prompts, and identify their own,
additional contributions.
>> 
>> Will it help them learn? When you are aware of the issues, it may be
quite useful. It may be especially useful to teach them to specify their
code carefully and completely, and to ask questions in the right way. Test
cases are crucial.
>> 
>> How will it affect what we do as instructors? I don't know. Really.
>> 
>> And the future? I am not pleased to extrapolate to a job market in 
>> which they compete with knowledge workers who work 24/7 without 
>> benefits, vacation pay, or even a salary. They'll need to rethink the 
>> value of their investment in an academic education. We'll need to 
>> rethink what we do to provide value above and beyond what AI's can 
>> do. (Nb. all of the arguments I hear about why humans will always be 
>> better etc. are easily debunked, but that's even more OT :-)
>> 
>> --------------------------------------------------------
>> 
>> If you have thoughts to share how your institution is thinking about
academic integrity in this situation, or creative ideas how to integrate
this into teaching, I'd love to hear from you.
> 
> *NEVER* let the AI misleading the students! ChatGPT gives you 
> seemingly sound but actually *wrong* code!
> 
> ChatGPT never understands the formal abstraction behind the code, it 
> just understands the shallow text pattern (and the syntax rules) in 
> the code. And it often gives you the code that seemingly correct but 
> indeed wrongly output. If it is used with code completion, then it is 
> okay (just like github copilot), since the coder need to modify the 
> code after getting the completion. But if you want to use ChatGPT for 
> students to query information / writing code, it is error proning!

______________________________________________
R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.