Accuracy, Hallucination and Reliability

In July 2025, Australia’s Department of Employment and Workplace Relations published an independent assurance review of the IT system it uses to apply automated penalties in the welfare system. The review came from Deloitte Australia, under a contract worth 440,000 Australian dollars. Weeks later a Sydney University researcher of health and welfare law, Chris Rudge, told journalists the report was “full of fabricated references”. It cited academic work that does not exist and quoted a federal court judgment with words the judge never wrote. A corrected version followed, this time disclosing that a generative AI tool built on Azure OpenAI had been used in drafting. Deloitte agreed to repay the final installment of its fee. The department said the findings and recommendations had not changed¹.

Nothing about the original report looked wrong. It was long, structured, footnoted and written in the confident register of professional assurance. That is the point. The errors survived drafting, review and publication because they looked exactly like the correct parts. An answer that says “I don’t know” invites someone to check. An answer that is fluent, specific and wrong slips past the check.

Fluent, wrong and expected

The AI Risk Landscape showed that risk depends on what the business allows a system to do with its answers. This chapter looks inside the first family on that map: why AI systems produce wrong answers, why those answers are so convincing, and how to build a system that catches them.

Start from a settled fact. Every AI system will sometimes be wrong, and no vendor can promise otherwise. The useful question is not whether the model errs but whether the system around it makes important errors less likely, detects them when they happen and stops them before they reach a customer, a payment or a decision. NIST’s AI Risk Management Framework names “valid and reliable” as a property of trustworthy AI, and defines reliability as the ability to perform as required, without failure, for a given time and under given conditions². Conditions are the key word. Reliability belongs to a system doing a job, not to a model in a lab.

Read the chain from left to right and you can see where a wrong answer is born. The question may be ambiguous. The system may retrieve the wrong document, an old version or a page it cannot read. The model may fill the gap with something plausible. And a missing check may let the result flow straight into action. Each link is also a place where the error can be caught, which is why the rest of this chapter walks the chain.

What a hallucination is

A hallucination is output that presents unsupported, invented or incorrect content as if it were valid: an invented fact, a citation to a paper that does not exist, a policy clause that was never written. NIST’s profile for generative AI prefers the word confabulation and lists it among the risks unique to, or made worse by, generative AI³. Researchers distinguish two kinds. An intrinsic hallucination contradicts the source the system was given, such as a summary that reverses a contract term. An extrinsic one adds content that the source cannot support, such as the fabricated court quote⁴.

The cause lies in what a language model does. It generates likely sequences of words from patterns learned in training. It is not a database of verified facts, and nothing in the generation step checks the words against the world. Plausible language and verified truth usually coincide, which is why these systems are useful. When they come apart, the output keeps its fluency.

This is why OWASP’s list of the main security risks in language-model applications pairs misinformation with overreliance: people trust credible-sounding output, skip the verification and build it into decisions⁵. A typo warns the reader. A fluent, specific, wrong answer does the opposite.

Why models guess

If models can be uncertain, why do they not say so? A 2026 paper in Nature gives an uncomfortable answer: we taught them not to. Adam Tauman Kalai and colleagues show that the way models are scored rewards guessing. Most of the benchmarks they reviewed grade an answer as right or wrong, and “I don’t know” earns the same zero as a wrong answer. Under that rule, a model that always guesses beats one that admits uncertainty, just as a student who guesses on every multiple-choice question outscores one who leaves blanks⁶.

The paper’s own example makes the trade visible. On SimpleQA, a set of short factual questions, two models from the same developer scored almost the same on accuracy. They were very different on errors.

The model that guesses is right on 20.6 percent of questions and wrong on 76.8 percent. The model that abstains is right on 16.0 percent and wrong on only 20.8 percent, because it declines the rest. A leaderboard that reports accuracy alone ranks the guesser first. For an enterprise process, the second model is far easier to make reliable, because its failures announce themselves. The authors propose scoring that states the penalty for an error up front, so that abstaining becomes the rational choice when stakes are high⁶.

The practical lesson is to ask for three numbers, not one: how often the system is right, how often it is wrong, and how often it declines. Accuracy alone tells you very little about the second.

Accurate on what?

The second trap is the single accuracy figure. Accuracy is how often a system produces the right result for the task being measured, and the task changes what “right” means.

Even for one task, an average can mislead. NIST’s framework says accuracy measurements should always be paired with realistic test sets that represent the conditions of expected use, and that results can be broken down by data segment². Medical-imaging researchers gave the failure a name, hidden stratification: models with strong overall scores performed poorly on important subgroups of patients that nobody had separated out in training or testing⁷. The average was true. It was not about the cases that mattered.

Two questions catch much of this. First, who wrote the test, and does its mix of questions match how people will really use the system? Second, what is the score for each important slice of work, not only overall? A test that under-samples one process cannot reveal concentrated failures in it. The story later in this chapter shows how.

Grounded is not the same as true

Many enterprise assistants first retrieve documents and then answer from them. RAG and Enterprise Knowledge — Executive Mental Model explained how this retrieval-augmented design works and why it beats retraining⁸. For reliability, two properties need to be kept apart. Factuality asks whether an answer is true. Groundedness asks whether it is supported by the source the system was supposed to use. Ask for the company’s travel reimbursement limit and common practice is irrelevant: the only right answer is the one in the current policy.

Grounding helps, and it does not end the problem. The retriever can fetch the wrong document, an outdated version or a scanned page whose text it cannot read, and the model will still write a coherent answer. The legal research tools in What Exactly Is Artificial Intelligence? showed that retrieval narrows the gap without closing it⁹. One large test of everyday assistants points the same way: in 2025, journalists at public broadcasters in 18 countries checked 2,709 news answers from four assistants.

Nearly a third of those answers, 31 percent, had serious sourcing problems: claims the cited source did not support, no source at all, or sourcing that could not be verified¹⁰. A citation is something to check, not proof. It can point to the wrong source, an old version or a real document that does not contain the claim.

Confidence, consistency and the right to say “I don’t know”

Many systems report a confidence score. It helps only if it is calibrated, meaning that answers given with 90 percent confidence are right about 90 percent of the time. That cannot be assumed. Modern neural networks are often overconfident even as they become more accurate¹¹. Before anyone relies on a confidence threshold, test it on your own work.

Consistency is a second signal. Ask a generative system the same question twice and the answers can differ. That is a nuisance where one answer is required, and it is also useful information: when repeated answers disagree in meaning, the system is probably making something up. Researchers have turned this into a hallucination detector that samples several answers and measures how far they agree¹². That is a result on research benchmarks; it has not yet been shown to cut errors in business deployments, so treat it as a promising capability. Where a task needs one fixed answer, such as a tax calculation, a date check or an ID match, give it to rules or ordinary software and keep the model for drafting, summarizing and explaining.

An underrated reliability feature is the right to decline.

“What is the approval limit?” Approval for what, for whom, in which region? A design that must always answer will guess. A design that can answer, ask or escalate will sometimes frustrate users and will far more often keep a wrong answer out of the process. The Kalai results put a number on the trade, and it favors the system that knows when to stop.

Checks the model cannot talk its way past

One of the strongest reliability patterns is also the simplest: let the model do the flexible work and let something else verify it. Take an illustrative invoice. An assistant reads it and reports a total of 125,000; a rule confirms that the line items add up to 125,000. An assistant quotes a value from a specification; a second step confirms that the value appears in the cited document and that the document is the current version. These checks are cheap and cannot be argued with, because they do not read the answer’s tone.

People are checks too, but only real ones. A reviewer adds reliability only with the evidence, the time, the expertise and the authority to override. Otherwise review becomes a signature on whatever the system produced. Why that happens, and how to design oversight that works, is the subject of Human Oversight and AI Incidents. For this chapter, the point is narrower: place automatic checks wherever an error can be detected mechanically, and save human attention for what only judgment can catch.

Story: a quality engineer’s ordinary day

The following is an illustrative composite, not a documented case. A global manufacturer of industrial equipment runs plants on three continents. Last year its engineering function launched an assistant that answers questions about internal procedures and supplier specifications. Before launch, a team at headquarters wrote 200 test questions and checked the answers. The assistant got 194 right: 97 percent. Plants adopted it quickly.

Follow one supplier-quality engineer through one day at one plant.

At ten past seven a shipment of castings arrives. She asks the assistant for the hardness range in the supplier’s specification. The answer is crisp, gives a range and cites the specification by number and section. Her inspection results fall inside it, and at half past eight she releases the batch to the line. Before lunch a buyer asks about a coating thickness and she answers the same way. At half past three she signs the receiving report. Nothing hesitated and nothing warned her. Engineers in other plants had days just like it.

Three weeks later, an engineer at a sister plant opens the specification itself to settle a dispute and finds a different range. The tables in many supplier specifications were scanned images. The assistant’s document pipeline could read the text around them but not the numbers inside them, so the model received a section heading and no values. It did what models are scored to do. It produced a plausible range typical of that material and cited the section it had been given. The citation was real. The number was not in it.

Why did 97 percent not reveal this? Because the test barely sampled the work that failed.

The headquarters team wrote its questions from procedures and narrative text, which it knew well. Only 10 of the 200 asked for a value from a table, and 4 of those 10 were wrong. With 2 other errors, the total came to 6 wrong and 194 right. The four table failures looked like scattered noise. (These figures are part of the illustration.) On the plant floor, though, about 30 percent of questions asked for a value from a table. Weight the test results by real use, 70 percent at 99 percent and 30 percent at 60 percent, and the system was right on about 87 percent of plant questions, with more than nine in ten of its errors in exactly the work the plants relied on most.

Was the model inaccurate, or was the system unreliable? Look at the chain. The question was clear. The source was the right document but unreadable. The model guessed instead of declining. No check confirmed that the quoted range appeared in the cited section. And the evaluation did not represent the work. The fixes follow the same chain: a test set drawn from logged plant questions and reported by slice, a rule that the quoted value must appear in the source text, abstention when a section cannot be read, and an owner for the specification library. None of them requires a better model. All of them are management decisions, and the errors belonged to the organization that ran the system.

What this means for leaders

Four habits follow. First, ask “accurate on what?” before accepting any score. Insist on test questions that match real use and on results broken down by the slices of work that matter. Second, ask for three numbers: right, wrong and declined. A system that declines honestly is easier to make reliable than one that guesses well. Third, put checks where the model cannot talk past them: arithmetic, quotations, source versions and limits. Fourth, reward the system and the people for saying “I don’t know”, and design the escalation path before launch rather than after the first incident.

Check yourself

1. If the assistant cites a source, the answer is supported by that source.MythFact
2. Grounding answers in your own documents ends hallucination.MythFact
3. A model with slightly lower accuracy can be the more reliable choice.MythFact
4. A system can score 97 percent in testing and fail one important process badly.MythFact
5. A high confidence score means the answer is probably right.MythFact
6. An assistant that asks a clarifying question has failed.MythFact

Reflection: find the untested slice

What comes next

A system can be accurate and still be unfair. Errors that are rare overall can fall disproportionately on one group of people, in the same way the table lookups hid inside the 97 percent. The next chapter, Bias and Fairness, looks at where bias comes from and why apparently neutral systems can produce unequal outcomes.