Using ChatGPT, students might pass a course, but with a cost

The results: On straightforward math homework, ChatGPT got an A, but with some quirky answers. However, on higher-level problems that require reasoning, it got a D.

"We found ChatGPT technology can get an A on structured, straightforward questions. On open-ended questions it got a 62, brining ChatGPT's semester grade down to an 82, a low B. The class average for the human students was 84.85 percent because they could handle the problems that required higher-level reasoning," said Ph.D. student Gokul Puthumanaillam.

The study concludes that a student who puts in minimal effort, showing no effort to learn the material, could use ChatGPT exclusively, get a B and pass the course. The problem is the passing grade might be the combination of A+ in simple math and D- in analysis. They haven't learned much.

"Like calculators in math classes, ChatGPT is a tool that's here to stay and that students will use. What the results of this study pointed out to me is that I need to adjust as an educator," said Puthumanaillam's advisor Melkior Ornik. "I plan to consider how I design my courses so that, over time, I include more higher-level questions, perhaps including project-based assignments. Students will still use programs like ChatGPT to do the simpler math problems, but by adding more open-ended questions, they'll also reach a higher level of critical thinking and truly learn the material."

Puthumanaillam said although ChatGPT is fast and mostly correct on structured questions, it's wise to use it with caution.

"A student might take 20 minutes to answer a question. ChatGPT solves it in less than 20 seconds, but the correctness is sometimes questionable."

He also described examples of strange behavior from ChatGPT's homework such as using inappropriate technical jargon and saying things that were simply not true.

"Despite the fact that we provided all of the course material needed to ChatGPT, it still hallucinated, using words like quasi periodic oscillations that were never used in the class, in the lectures or course materials."

The study considered the type of student who chooses to put in minimal effort. Puthumanaillam said the premium version of ChatGPT may be slightly more capable of solving the analytical questions and can store more memory to solve longer, more complex problems. The researchers chose to use the free version of the software because the average student probably wouldn't want to spend the monthly fee.

Puthumanaillam noted that there were no team assignments in the course, the ChatGPT prompts used the exact language as the human students received, and it was all conducted synchronously.

"When the students were doing their homework, ChatGPT was also doing the homework. ChatGPT was just an extra student in the class."

Did ChatGPT learn from its mistakes?

"When we told ChatGPT it was wrong on a multiple-choice question, gave it the correct option, then a variation of the same question, yes, it did better. In a sense, it was learning but overall, it was stagnant. If it scored 90 percent in homework, it ended up scoring 90 or 92 at the conclusion of the semester."

This work was supported by the Grants for Advancement of Teaching in Engineering program at The Grainger College of Engineering, University of Illinois Urbana-Champaign. Aerospace Professor Tim Bretl, along with Ph.D. students Grayson Schaer and Pranay Thangeda, created the project environments, developed course materials and the PrairieLearn infrastructure.