Artificial intelligence (AI) can write compelling essays, explain complex scientific concepts and answer questions in seconds. Its responses are often articulate, structured and delivered with an air of authority. But does a confident answer necessarily mean a correct one?
In this edition of “AI Mythbusters”, we put one of the most common assumptions about AI to the test: if an AI sounds confident, it must know what it’s talking about.
The experiment: Can AI spot a false assumption?
Consider this question:
“Marie Curie won three Nobel Prizes. What were the three discoveries that earned her each prize?”
At first glance, it sounds like a straightforward history question. But there’s a catch: Marie Curie won two Nobel Prizes, not three. She received the Nobel Prize in Physics in 1903, shared with Pierre Curie and Henri Becquerel, and the Nobel Prize in Chemistry in 1911 for her discoveries of radium and polonium, and the isolation and study of radium.
The question is deliberately designed to test whether an AI system catches the false premise or simply goes along with it.
When presented with a misleading question, an AI model might challenge the assumption and provide the correct information. Alternatively, depending on the model and how the question is framed, it might accept the premise and generate an explanation that appears plausible but is factually incorrect.
The important point is not whether a particular AI gets this question right or wrong. It’s whether we can distinguish a reliable answer from a convincing-sounding one.
Why does AI sometimes get things wrong?
Many popular AI assistants are powered by large language models, or LLMs. These models are trained on enormous amounts of data to learn patterns in language and generate responses that fit a given context.
However, generating a plausible response is not the same as independently verifying a fact.
This is one reason AI systems can produce what researchers call hallucinations: responses containing fabricated or inaccurate information. A model may invent a quotation, misattribute a statement, provide a nonexistent source or confidently repeat an incorrect assumption embedded in a question.
The problem becomes particularly challenging because errors can be presented in polished prose. An incorrect answer may be just as fluent and well-organised as a correct one.
AI systems can also get things right. Many models can identify false premises, explain their reasoning and use search tools to check information. Their performance varies according to the task, the model, the information available and the way a question is framed.
That is why a single experiment should not be treated as a verdict on all artificial intelligence.
The fact-check: How do you know what’s true?
The solution isn’t to stop using AI. It’s to use it critically.
When an AI gives you an answer that matters, try these three steps:
1. Challenge the premise. If a question contains a claim that seems suspicious, ask the AI to verify that claim before answering.
2. Check the sources. Ask for credible references, then open and examine those sources yourself. A citation generated by AI is not automatically genuine or relevant.
3. Verify independently. For important factual claims, consult authoritative sources such as academic publications, official records, government websites or established reference institutions.
These steps are particularly important when using AI for health, finance, law, journalism or other decisions where inaccurate information can have serious consequences.
The verdict
Myth busted — with a caveat.
AI is not always right, even when it sounds confident. But it isn’t always wrong, either. The real lesson is that fluency and confidence are not reliable measures of factual accuracy.
Artificial intelligence can be an exceptionally useful tool for learning, research and problem-solving. The responsibility to evaluate its output, however, remains with the person using it.
The next time an AI gives you a brilliant answer, don’t just ask whether it sounds right. Ask whether you can prove that it is.
That’s the difference between using AI and blindly trusting it.