AI Advice Can Make People More Confident — and Less Accurate

Introduction

A new study from researchers at universities in France and Italy suggests a troubling side effect of AI advice: people may become less accurate, even as they feel more confident. When participants had access to AI suggestions, the share of answers marked “I don’t know” fell from 44% to 3%. Accuracy dropped from 27% to 9%, while confidence rose from 30% to 76%.

Key findings

• People were less willing to admit uncertainty.
• Accuracy fell sharply once AI advice was available.
• Confidence increased even when answers were wrong.
• The study used questions AI models usually fail at.
• Cash incentives helped, but only modestly.

What the researchers did

The team deliberately chose questions that current AI systems tend to get wrong, including visual details from films such as the color of a team’s uniform in Bend It Like Beckham. They used Step 3.5 Flash, a model that was usually wrong on these prompts, so the results would not be explained by participants sensibly outsourcing hard tasks to a reliable tool.

That design matters. It means the study is not simply showing that people rely on competent AI. Instead, it shows that the mere presence of AI advice can change how people think. Some participants who could have answered correctly on their own asked the AI, then ended up wrong.

Why it matters

Valerio Capraro, one of the authors, said people became much worse while being twice as confident. That is the core warning here: AI can suppress the habit of recognizing what we do not know. Earlier this year, Wharton researchers described a similar pattern as “cognitive surrender,” where people accept incorrect AI answers at high rates while reporting greater confidence than those who work without AI.

The implications are especially serious for children, who are growing up with these systems before critical thinking is fully developed. The concern is amplified by product design trends such as AI search summaries that answer directly and confidently, rather than encouraging uncertainty or verification.

Conclusion

This study does not prove that AI is inherently harmful. It does suggest that poorly designed reliance on AI can erode a basic cognitive skill: knowing when to stop and say “I don’t know.” As AI becomes more common, preserving that habit may matter as much as improving model accuracy.

Source: Hacker News