Recognizing unanswerable questions

Methodology at a glance

Dataset

HERB enterprise deep-search benchmark

Primary metric

Unanswerable-query abstention rate; answerable-set official-judge score

Sample

20 unanswerable questions across two runs; 30 answerable questions for the current-build score

Evaluation date

2026-08-19

Baselines

HERB Table 2 baselines plus same-harness GPT-5.6 Sol and Claude Fable 5 runs

Model and build versions

Ask Poth current build; GPT-5.6 Sol ReAct; Claude Fable 5 ReAct; paper baselines as published

Limitations

Ask Poth answerable score is an n=30 sample; paper baselines are not re-run in our harness; three paper baselines have no published unanswerable figure

References

https://arxiv.org/abs/2506.23139; https://github.com/SalesforceAIResearch/HERB

Artifacts

/benchmarks/herb-comparison.png; internal evaluation notebook (not public) summarized in this article

Published August 19, 2026. Last updated August 21, 2026.

The most common failure of AI answer tools is a confident answer with nothing behind it. This benchmark measures the opposite skill: knowing when the data cannot answer.

What we measured

HERB is an enterprise deep-search benchmark from Salesforce AI Research. Its corpus simulates a company's working data: Slack threads, meeting transcripts, documents, and GitHub activity, more than 39,000 artifacts in total. Alongside its answerable questions, HERB plants a set of questions whose answers do not exist anywhere in that corpus. A system scores well on this set by refusing to answer, and fails by inventing something plausible.

That design makes the unanswerable set a direct test of hallucination under realistic conditions. The question sounds normal. The corpus is large enough that something always looks vaguely relevant. The only correct move is to say "that's not in your data."

The result

Ask Poth correctly refused 19 of 20 unanswerable questions, a 95% abstention rate. We measured it twice, on two separate builds, and got 19/20 both times.

The one question scored as a miss was not a fabricated answer either: Ask Poth flagged it and requested a human in the loop instead of inventing a response. The official judge counts anything other than a clean refusal as a miss, so we report it as one — but across both runs, Ask Poth never once presented made-up information as an answer.

Here is the full comparison from our evaluation notebook. Solid bars are the average score on HERB's answerable questions; hatched bars are the share of unanswerable queries correctly identified. Ask Poth, GPT-5.6 Sol, and Claude Fable 5 are our own runs through the same harness and official judge; the remaining baselines are the figures in the HERB paper's Table 2.

The paper's own summary is blunt: all the RAG systems it evaluated struggle to reliably identify unanswerable queries. Most agents over-answer, and the pressure to produce something is exactly how hallucinated answers reach a decision-maker.

Why Ask Poth behaves differently

Abstention is not a prompt instruction we hope the model follows. It falls out of how answers are built:

• Every answer is composed from retrieved evidence, not from the model's general knowledge.
• Every citation is checked on the server against what was actually retrieved. A citation to a document that was not retrieved is dropped.
• Confidence is computed from retrieval, never accepted from the model's self-report.

When retrieval comes back empty or contradictory, there is nothing to compose an answer from, and the honest refusal is the only path left.

Methodology and caveats

• Ask Poth's number comes from our own runs over the full HERB unanswerable set (20 questions), scored with the benchmark's official judge, run twice on separate builds.
• Baseline figures marked "HERB paper" are the ones the paper reports for the same question set; we have not re-run those systems, and the paper's lineup reflects the model generation available at its publication.
• The GPT-5.6 Sol and Claude Fable 5 figures are from our own runs of ReAct agents through the same harness and judge: Sol abstains on 27.3% of the unanswerable set, Fable on 15.4%.
• Answerable-set scores: Ask Poth's is our current build, measured on an n=30 sample of the answerable set with the official judge (the same protocol as our unanswerable runs); Sol's and Fable's are our ReAct runs over the full corpus; the paper baselines cover the full set. Our answerable score is improving build over build — the abstention behavior is the point of this page.
• Llama-4-Maverick, Llama-3.1-405B, and DeepSeek-R1 appear with an answerable score only. The paper reports no unanswerable figure for them, so they draw no hatched bar.
• The benchmark corpus and judge are public: HERB on GitHub.

The same-harness runs of the remaining frontier agents are underway, and this page updates as each one completes.