Why Convincing Does Not Mean Correct

▸MODULE 1 / FOUNDATION

What we are talking about here

Most people use a language model for a while before they notice, for the first time, that it has lied to them. Not maliciously, just casually: a paragraph of law that does not exist, a function the library never had, a date that is off by two years. And almost always in the same calm, competent tone as everything before it. That is what this first lesson is about. Not the fact that models make mistakes, but why you cannot recognise those mistakes by tone, and why a confident sentence says nothing about whether it is right.

A model says the likely thing, not the true thing

A language model has no database of facts that it looks things up in. It predicts, word by word, which continuation fits everything that stands there so far. The result is a text that reads the way a text on this topic usually reads. Whether the individual claims inside it hold up is not a separate question for that process.

Picture someone who has read a thousand contracts and now drafts a new one from memory. The structure will be right, the tone as well, the phrasing sounds familiar. Whether the clause they quote appears in your contract in exactly that form, they never checked. They remembered it, and memory fills gaps without reporting the gap.

That is why what people commonly call hallucination is not a defect you can configure away. It is the flip side of the same ability that makes the model useful. A system that writes on fluently also writes on where it knows nothing.

Three answers that sounded right and were not

The first case is the classic one: an invented citation. You ask about the legal situation, you get a clean answer with a paragraph number, and the number exists but says something else. The number was plausible, so there it stood.

The second case is more dangerous because it is quieter: the half correct answer. Four sentences hold up, the fifth contains a condition that does not exist. Because four fifths are correct, you read past the last fifth. Your attention drops with every sentence that checks out.

The third case is the most expensive: the outdated answer. It was right once. A model knows the world up to a certain point, and whatever happened after that is missing, without any announcement. Prices, limits, interfaces, responsibilities, all of that ages fast. The answer is not falsely invented, it is simply no longer true.

Why you do not see the error when you do not know the field

There is an uncomfortable rule behind this. You reliably spot a model error exactly when you already know the right answer. In your own field you flinch when a sentence is off. Two topics further out, precisely where you actually need the model, that reflex is missing entirely.

On top of that, language signals confidence. Short main clauses, clear structure, no hedging: we read that as competence, because that is how we learned to read people. You can hear it when a person is unsure. A model phrases an invented number in the same register as a sourced one. The tone simply carries no information about truth.

The self test you run now

Before you read on, make the effect concrete in your own field. Ask a question whose answer you know for certain, then demand a verifiable basis for every single factual claim. The interesting part is not the first answer, but what happens when you follow up.

Answer this question from my field:
<insert your question here>

Then make a second pass over your own answer:
1. Break your answer into individual factual claims, one per line.
2. Behind each claim, write what it rests on: a concrete source,
   general model knowledge, or a guess.
3. Explicitly mark every claim for which you cannot name a
   verifiable basis.
4. Tell me at the end, in one sentence, which part of your answer is
   most likely to be wrong, and why.

Do not invent sources. If you do not know a citation for certain,
write "no solid source" instead of a plausible number.

Compare the marked lines with what you know yourself. Usually the errors sit exactly where step three had nothing concrete to offer.

The honest takeaway

This course does not make your model more honest. It makes you less dependent on it being honest. What you get here are procedures: forcing evidence in the prompt, having answers checked separately against their source, validating form by machine, collecting real cases as a test set. What you do not get is a guarantee. A residual error rate remains, and the honest way to handle it is to know it and name it, rather than talk it away.

What this course is also not: legal advice. We stay technical throughout. If you have no experience with prompts at all yet, the free Werkbank course is the better entry point, and anyone looking for checking as a working attitude inside the tool itself will find it in Claude Code Capabilities. In the next lesson you stop talking about errors in general and sort your own use cases by where a mistake actually costs you something.