As AI tools become daily workhorses for generating reports and drafting analyses, they may be changing more than how we work. They may also be changing how we judge our own understanding of the material, according to new research from Hayden Noel (right), clinical professor of business administration at Gies Business; Erica Lee, a doctoral student in marketing; Adam King, Director of Innovation and Transformation; and Julia Sabin, Senior Associate Manager of Academic Innovation.
“The question we started with was, ‘Why are we so confident in the work we do when we use AI?’” said Noel. “We thought greater reliance on AI might create greater overconfidence. What we found was more complicated.”
The group released their findings in the paper, "The Impact of Fluency and Perceived Source on Confidence and Comprehension in AI-Generated Business Content."
The Fluency Effect
The research began with the polish of AI-generated content. Tools like ChatGPT and Claude tend to offer answers that are clean, easy to understand, and quick to process. That ease can make it difficult to separate processing information from actually learning it.
To illustrate the dynamic, Noel points to a tech trap most of us fall into outside the office.
“It’s called cognitive offloading, and we do it every time we use a GPS navigation system like Waze,” said Noel. “The GPS can get you where you need to go, but that doesn’t mean you learned how to get there. Take the GPS away the next day and you may realize you didn’t pay enough attention to the landmarks along the way.”
An initial pilot study by Noel and Lee provided support for their original concern. Participants read a fluent business explanation labeled as AI-generated, rated how well they understood the material, and then completed a comprehension assessment. Participants perceived their understanding to be higher than their objective performance suggested.
But when Noel and Lee expanded the research, they found something they did not expect.
In a subsequent study of 249 participants, they compared people who answered knowledge questions without assistance – learned from reading information – to those who received AI assistance. Both reading and AI assistance improved accuracy. Contrary to the researchers’ original expectations, however, AI users were not more overconfident about their performance.
Instead, AI-assisted participants performed about as well as those who learned through reading but reported lower perceived understanding.
Even more surprising was how little AI assistance it took to see that shift.
“We expected that greater reliance on AI might produce a larger effect,” said Noel. “But that’s not what we found. Even limited AI assistance was associated with lower perceived understanding, and using more AI didn’t appear to make the effect stronger.”
In fact, the pattern appeared even among participants who reported using AI for only a few questions. Additional exposure to AI did not increase the effect.
“That’s the part that really caught our attention,” said Noel. “We tend to think about AI reliance as a sliding scale, but our results suggest it may not be that simple. The difference between no AI and some AI may be more important than the difference between a little AI and a lot of AI.”
Board Room Reality Check
That finding matters well beyond the classroom.
Imagine an employee preparing for a critical client presentation. Most of the work may be their own, but they turn to AI for help answering a difficult question or developing part of the analysis. The resulting work is polished, accurate, and ready for the meeting.
But what happens when the client asks a question that wasn’t anticipated?
“That’s where understanding becomes different from simply producing the right answer,” said Noel. “AI can help you get to a strong answer, but in the board room you may have to explain why that answer makes sense, defend your reasoning, or apply it to something you haven’t seen before.”
Noel explained that the concern is not simply that AI makes people overconfident. The larger study did not find evidence of that. Instead, the findings raise questions about what happens to our understanding once AI becomes part of the process. For educators and managers, that distinction matters.
“If the quality of a finished assignment, report, or presentation is the only measure of success, it may be difficult to know what someone can actually understand, explain, and apply independently,” said Noel.
What Comes Next
The unexpected finding has raised new questions for Noel and Lee. Their next phase of research will continue to examine the relationship between AI use, perceived understanding, and actual knowledge, particularly why even limited AI assistance appears to change how people evaluate what they know.
“The big question for us now is why such a small amount of AI use can produce this shift,” said Noel. “We expected the effect to grow as people relied more heavily on AI, but that isn’t what we found. Understanding why is where the research goes next.”
Noel and Lee hope the research can help educators and organizations develop better guidelines for using AI while preserving the thinking and understanding people need to apply knowledge on their own.
For Noel, the goal is not to discourage people from using AI. It is to understand how to use it well.
“AI is going to be part of how we learn and work,” said Noel. “The question is how we use it in a way that helps us think better, rather than simply getting us to an answer faster.”