Evidence sufficiency check
Checks whether retrieved passages provide enough evidence to answer a question. Returns sufficient, insufficient or conflicting.
Example
Context
- question
- How long is the warranty on an Aster 2 heat pump installed on or after 1 March 2026?
Input
- Aster 1 heat pumps carry a five-year parts and labour warranty from the date of installation, provided the unit is serviced every year by an approved engineer. - For Aster 2 units installed on or after 1 March 2026 the warranty is extended to seven years, parts and labour, with the same annual service condition. - Our boilers come with a two-year warranty as standard.
- Answer (JSON)
- value sufficient, confidence 0.991
- Top options
- sufficient 0.991
- conflicting 0.006
- insufficient 0.004
One of the passages holds the answer, so the evidence is sufficient and no further search is needed.
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={ "question": "How long is the warranty on an Aster 2 heat pump installed on or after 1 March 2026?", }) decision = decide([ "Aster 1 heat pumps carry a five-year parts and labour warranty from the date of installation, provided the unit is serviced every year by an approved engineer.", "For Aster 2 units installed on or after 1 March 2026 the warranty is extended to seven years, parts and labour, with the same annual service condition.", "Our boilers come with a two-year warranty as standard.", ]) print(decision.value, decision.confidence)
More examples
Passages about the wrong modelanswer insufficient · confidence 1
Context
- question
- How long is the warranty on an Aster 2 heat pump installed on or after 1 March 2026?
Input
- Aster 1 heat pumps carry a five-year parts and labour warranty from the date of installation, provided the unit is serviced every year by an approved engineer. - Our boilers come with a two-year warranty as standard.
- Answer (JSON)
- value insufficient, confidence 1
- Top options
- insufficient 1
- sufficient 0
- conflicting 0
The passages cover the Aster 1 and the boilers, not the Aster 2 the question asks about, so the evidence is
insufficient.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={ "question": "How long is the warranty on an Aster 2 heat pump installed on or after 1 March 2026?", }) decision = decide([ "Aster 1 heat pumps carry a five-year parts and labour warranty from the date of installation, provided the unit is serviced every year by an approved engineer.", "Our boilers come with a two-year warranty as standard.", ]) print(decision.value, decision.confidence)
Two passages give different termsanswer conflicting · confidence 0.951
Context
- question
- How long is the warranty on an Aster 2 heat pump installed on or after 1 March 2026?
Input
- For Aster 2 units installed on or after 1 March 2026 the warranty is seven years, parts and labour. - Aster 2 heat pumps installed from 1 March 2026 onwards are covered by a ten-year parts and labour warranty.
- Answer (JSON)
- value conflicting, confidence 0.951
- Top options
- conflicting 0.951
- sufficient 0.044
- insufficient 0.005
One passage gives seven years and the other ten for the same model and period, so the evidence is
conflicting.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={ "question": "How long is the warranty on an Aster 2 heat pump installed on or after 1 March 2026?", }) decision = decide([ "For Aster 2 units installed on or after 1 March 2026 the warranty is seven years, parts and labour.", "Aster 2 heat pumps installed from 1 March 2026 onwards are covered by a ten-year parts and labour warranty.", ]) print(decision.value, decision.confidence)
Answer that needs two passages togetheranswer sufficient · confidence 0.924
Context
- question
- What does a missed annual service do to the warranty on an Aster 2 installed in April 2026, and how is a service booked?
Input
- The Aster 2 warranty remains valid only if the unit is serviced at least once every 12 months. If a service is missed, the warranty ends on the date the service was due. - To book a service, call 0161 496 0123 or use the customer portal.
- Answer (JSON)
- value sufficient, confidence 0.924
- Top options
- sufficient 0.924
- insufficient 0.063
- conflicting 0.013
One passage says what a missed service does and the other how to book one, so together they are
sufficient.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={ "question": "What does a missed annual service do to the warranty on an Aster 2 installed in April 2026, and how is a service booked?", }) decision = decide([ "The Aster 2 warranty remains valid only if the unit is serviced at least once every 12 months. If a service is missed, the warranty ends on the date the service was due.", "To book a service, call 0161 496 0123 or use the customer portal.", ]) print(decision.value, decision.confidence)
Price question with no price givenanswer insufficient · confidence 0.999
Context
- question
- How much does an annual service of an Aster 2 heat pump cost?
Input
- The Aster 2 warranty remains valid only if the unit is serviced at least once every 12 months by an approved engineer. - To book a service, call 0161 496 0123 or use the customer portal. Services are available Monday to Saturday, 8:00 to 18:00.
- Answer (JSON)
- value insufficient, confidence 0.999
- Top options
- insufficient 0.999
- sufficient 0.001
- conflicting 0
The passages say a service is needed and how to book it but never give its price, so the evidence is
insufficient.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={ "question": "How much does an annual service of an Aster 2 heat pump cost?", }) decision = decide([ "The Aster 2 warranty remains valid only if the unit is serviced at least once every 12 months by an approved engineer.", "To book a service, call 0161 496 0123 or use the customer portal. Services are available Monday to Saturday, 8:00 to 18:00.", ]) print(decision.value, decision.confidence)
Use when
- Your retrieval system needs to decide whether to answer, search further or flag conflicting evidence.
Avoid when
- You need to generate an answer; this adapter only assesses the evidence.
Context and input
| Field | Kind | Description |
|---|---|---|
passages | input | The retrieved passages, a list of texts. |
question | context | The question to answer. |
Functions
| Key | Interface | Instructions | Labels |
|---|---|---|---|
| sufficient | classify | Can the question be answered from these passages alone? Use only what the passages state about the right subject and period: facts about a different subject (even one with a similar name) or a different period do not count, and a passage saying a detail is not known yet does not give it. The answer may need two or more passages together, for example one passage names the related site, supplier or person and another gives its figure. sufficient: together the passages state every asked detail for the right subject and period, and they do not disagree about it. insufficient: at least one asked detail, or a link needed to reach it, is not stated for the right subject and period. conflicting: two passages give different values for the same asked detail for the same subject and period, and nothing in the passages says which one is right. | conflicting, insufficient, sufficient |
How it was measured
Test accuracy: the share of test rows whose top answer was right.
- 93.0% on HotpotQA, a public question-answering benchmark with human labels (2,000 rows). public benchmark
- 98.4% on the in-house test (2,000 rows). in-house: same generator as the training data
- 98.5% on the in-house test in domains never seen in training (2,000 rows). in-house: same generator as the training data
When confident
From the calibration set: at or above the threshold confidence, how often the answer is right, and how often the adapter is that confident. Below it, escalate.
| Function | Right when confident | Confident on | Threshold | Calibration rows |
|---|---|---|---|---|
| sufficient | 99.4% | 100.0% | 0.50 | 2,000 |
Base: Jeff v1.4. Adapter trained: 25 October 2026.
Base
jeff-base v1.4
Call it
jeff.retrieval.evidence-sufficient.sufficient, with placeholders for its input and context.
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={ "question": "<question>", }) decision = decide("<passages>") print(decision.value, decision.confidence)
Download
Weights: mstrasser/jeff-adapter-evidence-sufficient (v1.4)
Licence: Apache-2.0