Generative chatbots are being hailed as a game-changing solution for delivering personalized education at scale, but their accuracy limitations pose significant challenges, according to research by Assistant Professor Tiffany Li.

Personalized learning is widely regarded as an effective teaching method, yet resource constraints prevent its widespread application.

In traditional small classrooms, instructors can tailor lessons, engage with learners individually and adjust teaching to suit each student’s needs—a level of personalization that becomes increasingly difficult in large or online classes.

Pedagogical chatbots, powered by generative AI, are emerging as an alternative. They offer individualized, on-demand support, responding to students’ questions, addressing misconceptions in real time and doing so simultaneously for countless learners—something human instructors cannot match.

These chatbots are particularly useful for learners who prefer self-paced study or operate outside conventional classroom environments.

However, Li warns that chatbots are not without flaws. They sometimes produce incorrect, incomplete or fabricated information, which can adversely impact learners.

To examine how well students detect such inaccuracies, Li and her research team conducted a systematic study by developing a pedagogical chatbot for introductory statistics.

The chatbot was programmed to intentionally introduce factual errors, allowing researchers to observe whether learners could identify them. A total of 177 college students and adult learners participated, using the chatbot to work through practice problems while having access to textbooks and search engines.

They were instructed to report detected errors, with a monetary incentive provided to encourage verification.

The findings were striking. Despite having tools and incentives to verify answers, learners had only about a 15% success rate in reporting chatbot errors. These inaccuracies significantly affected learning outcomes.

When participants encountered erroneous chatbot responses, their accuracy on practice problems dropped to between 25% and 30%. In contrast, those in the no-error control group achieved an accuracy range of 60% to 66%.

The research team presented their findings in a paper titled “Can Learners Navigate Imperfect Generative Pedagogical Chatbots? An Analysis of Chatbot Errors on Learning,” at the Conference on Learning @ Scale.

They found that one reason learners struggled was their reliance on less effective verification strategies, such as trusting their prior knowledge to assess chatbot accuracy.

For learners with limited or incorrect knowledge, this approach proved unreliable, making them more vulnerable to accepting erroneous information.

The study highlights the promise of generative chatbots in transforming education, while also emphasizing the urgent need to address their accuracy and reliability to protect and enhance learning outcomes.