Skip to content
veri—ableMy research
All visual stories

EVALUATION / STORY 06

A benchmark is only as good as its questions.

A closer look at Epoch’s sample audit of Humanity’s Last Exam.

3 min readEvidence dated 11 Aug 2026
QUESTION SAMPLEEpoch AI
22 / 48

sampled questions with reported errors

Each dot is one sampled question. This is not the complete benchmark.

THE SHORT ANSWER

Look inside the score.

  1. Epoch reports substantial accuracy-altering errors in 22 of 48 sampled questions from the original Humanity’s Last Exam.
  2. The sample took six questions from each of eight categories. Epoch used AI-assisted review followed by manual checks.

Keep the limits in view.

This is Epoch’s audit, not a reproduction by Veri—able. The sample does not establish an exact error rate for all questions or apply to HLE-Rolling or HLE-Verified. Follow-up responses are not yet included.

Keep asking.

Ask about this evidence

Collection reviewed 30 September 2026. Initial research edition. Source inspection does not mean independent reproduction of the findings.