Skip to content

Worked examples

A retrieval pipeline that knows when not to answer

The problem

An e-bike maker's help assistant answers customers from its own documents: manuals, warranty terms, orders and prices. A retrieval pipeline finds passages and a language model writes the answer. When the passages do not hold the answer, the model writes one anyway, and the customer is told something the company never promised.

How Jeff helps

Four retrieval functions check the pipeline at each step, each in one fast call:

  1. Query routing picks the source to search.
  2. Passage relevance says whether a passage bears on the question.
  3. Evidence sufficiency says whether the passages kept are enough to answer. When they are not, the assistant says "I don't know" instead of guessing.
  4. Grounding check checks the written answer against the passages before it is sent.

The solution

Each step below is a live call; the answer under it is the one the API returned. Choose your language and whether to run it on your machine (free) or through the API.

Pick the source

A customer asks whether the warranty on their bike still covers the battery after two years:

Context

menu
- product_manuals: user and repair manuals for every Kestrel e-bike - warranty_terms: the warranty terms for each model, by model and purchase date - customer_orders: each customer's orders, deliveries and repairs (a database) - pricing: current prices, spare parts and offers

Input

Does the warranty on my Kestrel Volt 2 still cover the battery after two years?

Labels (the options)

product_manuals
User and repair manuals for every Kestrel e-bike
warranty_terms
The warranty terms for each model, by model and purchase date
customer_orders
Each customer's orders, deliveries and repairs (a database)
pricing
Current prices, spare parts and offers
none
No source on the menu holds what the query needs, or no lookup is needed
several
The answer needs facts from two or more of these sources
Answer (JSON)
value warranty_terms, confidence 0.975
Top options
  1. warranty_terms 0.975
  2. none 0.008
  3. customer_orders 0.008

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.retrieval.query-route.source", context={        "menu": "- product_manuals: user and repair manuals for every Kestrel e-bike\n- warranty_terms: the warranty terms for each model, by model and purchase date\n- customer_orders: each customer's orders, deliveries and repairs (a database)\n- pricing: current prices, spare parts and offers",    }, labels={        "product_manuals": "User and repair manuals for every Kestrel e-bike",        "warranty_terms": "The warranty terms for each model, by model and purchase date",        "customer_orders": "Each customer's orders, deliveries and repairs (a database)",        "pricing": "Current prices, spare parts and offers",        "none": "No source on the menu holds what the query needs, or no lookup is needed",        "several": "The answer needs facts from two or more of these sources",    })     decision = decide("Does the warranty on my Kestrel Volt 2 still cover the battery after two years?")     print(decision.value, decision.confidence)

warranty_terms: the question is about the warranty, so the pipeline searches the warranty terms.

Keep the passages that bear on the question

The search returns two passages: the Volt 2's warranty terms, and a passage on charging. The charging passage does not answer the question:

Context

source
warranty_terms: the warranty terms for each Kestrel e-bike model, by model and purchase date
question
Does the warranty on my Kestrel Volt 2 still cover the battery after two years?

Input

Charge the Volt 2 battery at room temperature with the supplied charger. A full charge takes about four hours.

Answer (JSON)
value no, confidence 0.995
Top options
  1. no 0.995
  2. yes 0.004
  3. partly 0

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.retrieval.passage-relevant.relevant", context={        "source": "warranty_terms: the warranty terms for each Kestrel e-bike model, by model and purchase date",        "question": "Does the warranty on my Kestrel Volt 2 still cover the battery after two years?",    })     decision = decide("Charge the Volt 2 battery at room temperature with the supplied charger. A full charge takes about four hours.")     print(decision.value, decision.confidence)

no: the pipeline drops it. For the warranty passage, the one that does answer, passage relevance also answered no (0.62, recorded on 4 November 2026), even after the passage was written more plainly. So the pipeline does not drop a passage on this function's answer alone: a passage it rejects goes to your language model to confirm before it is dropped.

Answer, or say "I don't know"

With the warranty passage kept, the evidence is enough:

Context

question
Does the warranty on my Kestrel Volt 2 still cover the battery after two years?

Input

- Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.

Answer (JSON)
value sufficient, confidence 0.721
Top options
  1. sufficient 0.721
  2. insufficient 0.26
  3. conflicting 0.019

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={        "question": "Does the warranty on my Kestrel Volt 2 still cover the battery after two years?",    })     decision = decide([        "Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.",    ])     print(decision.value, decision.confidence)

sufficient: the assistant writes an answer from it.

A second customer asks whether the warranty covers a battery damaged when the bike was left out in the rain. The passages say nothing about water damage:

Context

question
Does the warranty on my Kestrel Volt 2 cover a battery that was damaged when the bike was left out in the rain?

Input

- Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent. - Charge the Volt 2 battery at room temperature with the supplied charger. A full charge takes about four hours.

Answer (JSON)
value insufficient, confidence 0.994
Top options
  1. insufficient 0.994
  2. sufficient 0.005
  3. conflicting 0.001

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.retrieval.evidence-sufficient.sufficient", context={        "question": "Does the warranty on my Kestrel Volt 2 cover a battery that was damaged when the bike was left out in the rain?",    })     decision = decide([        "Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.",        "Charge the Volt 2 battery at room temperature with the supplied charger. A full charge takes about four hours.",    ])     print(decision.value, decision.confidence)

insufficient: the assistant says it does not know and offers to pass the question to the service team, instead of guessing.

Check the answer before it is sent

The assistant's answer to the first customer stays within the warranty passage:

Context

sources
- Warranty terms: Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.

Input

Yes. The Volt 2 battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent, so after two years it is still covered.

Answer (JSON)
value supported, confidence 0.554
Top options
  1. supported 0.554
  2. partly_supported 0.394
  3. contradicted 0.048

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.retrieval.ground.grounding", context={        "sources": [            "Warranty terms: Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.",        ],    })     decision = decide("Yes. The Volt 2 battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent, so after two years it is still covered.")     print(decision.value, decision.confidence)

supported, but with confidence 0.55, below the function's threshold of 0.63. Here the decision abstains, so with an escalation policy the question goes to your language model or a person before the answer is sent.

A different draft adds a promise the warranty terms do not make:

Context

sources
- Warranty terms: Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.

Input

Yes. The battery is covered for three years, and Kestrel will send you a free replacement battery by next-day delivery.

Answer (JSON)
value partly_supported, confidence 0.999
Top options
  1. partly_supported 0.999
  2. contradicted 0.001
  3. unsupported 0

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.retrieval.ground.grounding", context={        "sources": [            "Warranty terms: Kestrel Volt 2 warranty. The frame is covered for ten years. The battery is covered for three years from the date of purchase, against defects and against its capacity falling below 70 percent.",        ],    })     decision = decide("Yes. The battery is covered for three years, and Kestrel will send you a free replacement battery by next-day delivery.")     print(decision.value, decision.confidence)

partly_supported: the three-year cover is in the passage, the free next-day replacement is not. Send the draft back to be rewritten without it.

What you get

  • An assistant that says "I don't know" when its documents do not hold the answer, and that checks each answer before it goes out.
  • Confidence on every step, so an unsure answer (like the grounding check at 0.55 above) is escalated rather than trusted.
  • How the functions measure, from each adapter's page. Each has a public-benchmark figure below its in-house one: passage relevance 83.9% on SQuAD 2.0 (2,000 rows) against 99.5% in-house, and evidence sufficiency 93.0% on HotpotQA (2,000 rows) against 98.4% in-house. Query routing scores 99.3% on its in-house test (2,000 rows, from the same generator as the training data), and the grounding check 96.6% (4,160 rows, mixed sources). Expect the public figures, not the in-house ones, on documents unlike the training data.