Why LLMs Sound Sure Even When They are Wrong and How Distractors Help?

If you have used a large language model (LLM) for factual questions, you have probably seen this pattern: the model delivers an incorrect answer with the tone and confidence of someone who is absolutely sure they are right. For example, as illustrated in Figure 1, when asked “Who received the IEEE Frank Rosenblatt Award in 2010?”, a leading LLM confidently but incorrectly answers “Geoffrey Hinton” with a confidence of 93%, despite the correct answer being “Michio Sugeno”.

Figure 1: An instance from SimpleQA dataset where an LLM assigns high confidence to an incorrect answer.
Figure 1: An instance from SimpleQA dataset where an LLM assigns high confidence to an incorrect answer.

That mismatch — high confidence when correctness is low — isn’t just annoying. In high-stakes settings (healthcare, finance, legal workflows), it is dangerous because humans tend to trust confident outputs. In my TMLR paper, I studied this problem through the lens of calibration: Does the model’s confidence actually reflect its likelihood of being correct?

This article walks through:

  • what calibration really means (without heavy math),
  • what we found across 9 major LLMs and 3 QA datasets, and
  • a surprisingly effective intervention: structured distractors.

If this article helped you or inspired your own research, please consider citing the paper.

@article{
  chhikara2025mind,
  title={Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models},
  author={Prateek Chhikara},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2025},
  url={https://openreview.net/forum?id=lyaHnHDdZl},
  note={}
}

What is Calibration in plain English?

A model is well-calibrated if:

  • When it says “I am 80% confident,” it’s correct about ~80% of the time.
  • When it says “I am 30% confident,” it’s correct about ~30% of the time.

A model is overconfident if it routinely claims high confidence but gets many of those answers wrong. To quantify calibration error, we use a standard metric called Expected Calibration Error (ECE): lower is better; 0 is perfect calibration.

The key idea: what if we force the model to “consider the opposite”?

In human psychology, one way to reduce overconfidence is to explicitly consider alternative hypotheses (“consider the opposite”). Inspired by this, we tested whether LLMs become better calibrated if they must answer in a setting that includes plausible wrong options — i.e., distractors. So we compared two prompting regimes:

  1. Free-generation (baseline): The model produces an answer and a self-reported confidence score (0–100).
  2. Distractor-augmented (structured choices): For each question, we provide: 1 correct answer, 3 plausible but incorrect distractors, and shuffled these options. The model must pick from the list and give confidence. Distractors were generated to match the correct answer’s type (person/date/number/place), be plausible, and be distinct.

Experimental setup (what we evaluated)

We evaluated nine LLMs spanning:

  • different scales (≈8B up to very large closed models),
  • architectures (dense vs MoE),
  • alignment regimes (SFT vs RLHF).

We tested on three factual QA datasets:

  • SimpleQA (hard, short factoid questions),
  • FaVIQ (moderate difficulty),
  • TriviaQA (easier in this setup).

Correctness was judged using a consistent LLM-judge setup (GPT-4o-mini) to avoid instability from weaker judges.

Figure 2. Performance metrics of LLMs in the Normal (N) and Distractor (D) settings on the SimpleQA, FaVIQ, and TriviaQA datasets, including accuracy (correct), NOT_ATTEMPTED (na), ECE, and the number ofhelped (D_helped) and harmed (D_harmed) instances with their percentages.
Figure 2. Performance metrics of LLMs in the Normal (N) and Distractor (D) settings on the SimpleQA, FaVIQ, and TriviaQA datasets, including accuracy (correct), NOT_ATTEMPTED (na), ECE, and the number of
helped (D_helped) and harmed (D_harmed) instances with their percentages.
Figure 3. Reliability diagrams (RDs) showing calibration performance in N (purple) and D (yellow) settings on the SimpleQA dataset. (y-axis: actual accuracy, x-axis: predicted confidence)
Figure 3. Reliability diagrams (RDs) showing calibration performance in N (purple) and D (yellow) settings on the SimpleQA dataset. (y-axis: actual accuracy, x-axis: predicted confidence)

Findings of the Experiments

Finding 1: Overconfidence is widespread, even in strong models

On the hardest dataset (SimpleQA), even the strongest models are far from perfectly calibrated. For example, the paper notes GPT-4o achieves ~35% accuracy with non-trivial ECE in free-generation. The broader point: scale alone doesn’t eliminate miscalibration on hard factual queries.

Finding 2: Distractors massively improve accuracy and often reduce ECE

Adding structured distractors consistently boosts accuracy and generally reduces calibration error across models — especially on difficult datasets. A striking example on SimpleQA: GPT-4o-mini jumps from 8.46% → 47.43% accuracy and ECE drops from 0.750 → 0.320 in the distractor setting.

Finding 3: Bigger models get better-calibrated probabilities — not just better answers

Small models can gain a lot of accuracy when you give options, but their confidence estimates often remain misaligned. In contrast, larger models tend to produce more trustworthy confidence scores, especially after structured prompting. A simple intuition: small models learn factual patterns faster than they learn to know when they know and larger models improve more at self-assessment — confidence that tracks correctness.

Finding 4 (the nuance): distractors can hurt calibration on easy questions

This is where the story gets interesting. On TriviaQA (the easiest dataset here), some large RLHF-tuned models show slight ECE increases under distractors even though accuracy still improves. The paper attributes this to “confidence inflation” on already-easy examples: multiple-choice framing can push the model into overcommitting (becoming even more certain).

Figure 4: Performance (correct) of LLMs across different question types in both N (purple) and D (yellow) settings.
Figure 4: Performance (correct) of LLMs across different question types in both N (purple) and D (yellow) settings.

Finding 5: “Person” questions are the most stubborn failure mode

SimpleQA tags questions into Date/Number/Person/Place, and a consistent theme emerges: Person-based queries are the hardest. They are ambiguous (similar names, overlapping roles, contextual dependencies), and models often confuse related entities. Distractors help the most here (largest relative ECE drop across types), but person questions still remain a major calibration pain point across models.

Limitations (what we were careful about)

One important limitation: the distractors are generated using a fixed generator and the evaluation uses a fixed LLM judge for consistency. That reduces rubric drift but also means results are conditional on that generator/judge pairing. The paper mitigates this with plausibility checks and human spot-checking, and releases prompts/code for replication.

Closing thought: calibration is the missing reliability layer

Accuracy tells you how often a model is right. Calibration tells you whether you can trust the model when it says it’s right.

And the core message from the paper is:

  • Overconfidence is widespread in LLMs on factual QA.
  • Structured distractors can significantly reduce miscalibration and improve accuracy
  • But calibration remains nuanced — especially across difficulty and question type.

If you are interested in reproducing or extending the benchmark, the paper links to the public code repository.

GitHub - prateekchhikara/llms-calibration