How Confident Should You Be?

The problem with confidence words

Two people agree that an outcome is “likely”. One means about 60 percent. The other means about 90 percent. They act very differently, they both believe they were understood, and neither discovers the mismatch until something goes wrong.

This is not a hypothetical failure. It is the ordinary behaviour of unanchored confidence language, and it is why several analytic professions publish fixed scales that map confidence words to stated ranges.

The remedy is not to put a percentage on everything. It is to make three things explicit: how confident you are, what that confidence rests on, and what would change it.

What confidence should track

Confidence should follow the quality of your evidence, not the strength of your feeling or the fluency of the account you can give.

Four things reasonably raise it:

Source quality. Direct access beats secondhand. Documents and records beat recollection. A named party with something to lose beats an anonymous one.

Independence of corroboration. Two accounts that did not derive from each other are worth far more than five that did. This is the single most commonly overstated form of support.

Consistency across methods. When a survey, an administrative dataset, and a field observation point the same way, the chance that all three share the same error is lower.

A mechanism you can state. Being able to describe how the thing would happen constrains the claim and generates further predictions.

Three things reasonably lower it:

Scope stretch. The evidence supports a narrower claim than the one being made. This is the most common quiet failure.

Unexamined alternatives. You have not seriously considered other explanations.

Time. Evidence about a changing situation decays. A confident judgment based on a six-month-old dataset about a fast-moving question should not stay confident by default.

Three things should not affect it, though they usually do: how many times you have heard the claim, how fluently it was expressed, and how much you want it to be true.

Language that holds up

A defensible confidence statement has three parts: the claim, the confidence, and the basis.

Weak: “The programme is working.”

Better: “I am moderately confident the programme reduced missed appointments, based on a single before-and-after comparison at one site, with no control group. A comparison site or a second year of data would raise or lower that substantially.”

The second version is longer, and the length is doing work. It states the claim precisely, it says how sure, it names the evidence, and it says what would move it. Someone can now disagree with a specific part rather than with the sentiment.

If you use words rather than numbers, use them consistently, and say once what you mean:

• Low confidence. Plausible, but the evidence is thin, single-sourced, or indirect.
• Moderate confidence. Reasonable evidence, with a known gap or an unresolved alternative.
• High confidence. Multiple independent lines of evidence, a stated mechanism, and no substantial contrary evidence.

High confidence is not certainty, and it should still come with a statement of what would overturn it.

Saying what would change your mind

This is the part most people skip, and it is the most valuable.

Before you look at more evidence, write down what result would lower your confidence. Do it in advance, because doing it afterwards is not the same exercise. Afterwards, everything can be accommodated.

If you cannot name anything that would change your mind, one of three things is true. The claim may be a value judgment rather than a factual one, in which case evidence was never the right instrument. It may be so well established that the question is not live. Or you may be holding it for reasons other than evidence, which is worth knowing.

Practising calibration

Calibration improves with feedback, and feedback requires a written record.

A simple practice: for four weeks, write down five predictions a week about things that will resolve within a month. Ordinary things are fine. Attach a probability to each. At the end, group them by the probability you assigned and check how often each group came true.

If the things you called 70 percent happened about 70 percent of the time, you are calibrated in that range. If they happened 40 percent of the time, you are overconfident, which is the common direction. The specific number matters less than discovering that you have a pattern at all.

Two things calibration is not

It is not hedging. A calibrated statement is more specific than a vague one, not less. “Moderate confidence, based on one site, no control group” commits you to more than “it seems to be working.”

It is not treating all positions as equally uncertain. Calibration is the practice of matching confidence to evidence, which means being confident where the evidence is strong. Uniform doubt is as poorly calibrated as uniform certainty.

Summary

• State the claim precisely enough that it could be wrong.
• Say how confident you are, using words you have defined or a number.
• Name the evidence the confidence rests on.
• Say in advance what would raise or lower it.
• Keep a written record and check yourself against it later.

No external references were required for this page. The confidence bands described here are a simplified version of conventions used in several analytic professions, presented for general public use.

Next step

Practice in the Think Lab

Topics covered: Information ResilienceCritical Thinking