Why it sounds certain when it’s wrong

If you read one page on this site before touching any of this, make it this one. Not because AI is dangerous, but because almost every bad experience people have with it comes from the same misunderstanding — and once you see what the thing is actually doing, both the fear and the disappointment get a lot smaller.

What it is actually doing

A language model is a very large pattern-matcher trained on an enormous amount of text. Given what has been written so far, it produces what would plausibly come next, one small piece at a time. That is the whole mechanism. It is not looking anything up, it has no list of facts inside it, and it is not checking its answer against anything.

This is why it is so good at some things and so unreliable at others, and the split is not random.

The one idea to remember

It is not answering your question. It is producing text that looks like a good answer to your question. Most of the time those are the same thing, which is exactly what makes the exceptions hard to spot.

Why a wrong answer sounds just like a right one

Confidence is a property of writing, not of knowledge. The model learned what a confident, well-organised, authoritative answer looks like, and it produces that shape whether or not the content underneath is correct. There is no internal signal that gets weaker when it’s guessing.

Compare that with a person. When someone half-remembers something, you usually hear it: the pause, the hedge, the “I think it was around 2019?” Those cues are what we’ve all spent our lives using to calibrate trust. The model doesn’t produce them reliably, so the instinct you’ve relied on since childhood gives you no information here. That’s uncomfortable, and it’s worth naming rather than working around.

The things it invents, and why those specifically

The fabrications follow a pattern. They cluster around exactly the material that has a strong, recognisable shape but no strong presence in what the model was trained on:

Citations, sources and references. It knows precisely what a reference looks like — author, title, journal, year — so it can produce a perfect one that refers to nothing.
Quotations. A plausible sentence in the right voice is easy to generate. Whether the person said it is a separate question the model isn’t asking.
Numbers, dates and statistics. These have an obvious form and are easy to produce in the right shape and the wrong value.
Specifics about small or local things. A named business, a minor rule, a particular product version. The less something appeared in the training material, the more the answer becomes an educated impersonation of one.
Anything after its training stopped. Models are trained up to a date. Ask about something more recent and you may get a confident answer built entirely out of what would have been reasonable to expect.

Where it is genuinely reliable

The useful rule is about where the information is coming from.

Working on text you supplied

Reliable.

Rewriting, shortening, changing the tone, explaining a paragraph you pasted in, turning notes into a letter, translating, fixing grammar, drafting from your own bullet points. The material is in front of it, so there is nothing to invent.

Recalling things from memory

Check it.

Facts, dates, names, prices, legal or medical specifics, whether something exists, what a particular company’s policy says. Here it is reconstructing from patterns, and a confident reconstruction is indistinguishable from a memory.

This is also the reason the private document search in Chapter 01 is worth the setup. It moves questions from the right-hand column to the left by putting your actual documents in front of the model before it answers.

Three habits that cover almost everything

Ask yourself what happens if this is wrong. A wrong word in a birthday card costs nothing. A wrong dosage, deadline, or dollar figure costs a great deal. Scale your checking to that, not to how confident the answer sounded.
Check anything you would have to cite. If you’d repeat it to a doctor, put it in a letter, or tell a friend it’s true, spend the thirty seconds. Look up the source rather than asking the model whether it was right — asking again just generates another confident paragraph.
Notice when you’ve stopped reading carefully. Fluent text is restful, and after twenty good answers your attention drops. That’s the moment the twenty-first one gets through.

A note on “hallucination”

You will see this behaviour called hallucinating, which makes it sound like a malfunction — a machine that normally knows things having a funny turn. It isn’t. The model is doing exactly what it always does; the only difference is whether the plausible continuation happens to be true. Understanding that is what turns “sometimes it lies” into something predictable and manageable.

Why this should make you less anxious, not more

People who fear this technology and people who over-trust it are making the same mistake in opposite directions: both are treating it as a mind that knows things. It isn’t one. It is a remarkably good instrument for working with language, which is genuinely useful and genuinely limited, in ways you can now predict.

You don’t need to be vigilant about everything. You need to know which column you’re in.