jeff.agents.result-relevantv1.4.0
Tool result relevance
Checks whether a tool result answers the agent's current question: yes, partly or no.
Example
Context
- app
- Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices and clients and emails invoices on request.
- today
- Monday 2 November 2026
- sub_question
- What is the invoice number and total of Orbit Print's October 2026 invoice?
- call
- tool: find_invoice description: Finds the studio's invoice for a client and month. arguments: {"client":"Orbit Print","month":"2026-10"}
Input
invoice_id: INV-2026-0388 client: Orbit Print month: 2026-09 total: 1980 status: paid
- Answer (JSON)
- value no, confidence 0.954
- Top options
- no 0.954
- partly 0.045
- yes 0.001
The agent needed October's invoice and the tool returned September's, so the answer is no.
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.agents.result-relevant.relevant", context={ "app": "Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices and clients and emails invoices on request.", "today": "Monday 2 November 2026", "sub_question": "What is the invoice number and total of Orbit Print's October 2026 invoice?", "call": { "tool": "find_invoice", "description": "Finds the studio's invoice for a client and month.", "arguments": { "client": "Orbit Print", "month": "2026-10", }, }, }) decision = decide({ "invoice_id": "INV-2026-0388", "client": "Orbit Print", "month": "2026-09", "total": 1980, "status": "paid", }) print(decision.value, decision.confidence)
More examples
October invoice with its number and totalanswer yes · confidence 0.998
Context
- app
- Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.
- today
- Monday 2 November 2026
- sub_question
- What is the invoice number and total of Orbit Print's October 2026 invoice?
- call
- tool: find_invoice description: Finds the studio's invoice for a client and month. arguments: {"client":"Orbit Print","month":"2026-10"}
Input
invoice_id: INV-2026-0412 client: Orbit Print month: 2026-10 total: 2350 status: unpaid
- Answer (JSON)
- value yes, confidence 0.998
- Top options
- yes 0.998
- partly 0.001
- no 0
The result is October's invoice for Orbit Print with its number and total, everything asked, so the answer is
yes.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.agents.result-relevant.relevant", context={ "app": "Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.", "today": "Monday 2 November 2026", "sub_question": "What is the invoice number and total of Orbit Print's October 2026 invoice?", "call": { "tool": "find_invoice", "description": "Finds the studio's invoice for a client and month.", "arguments": { "client": "Orbit Print", "month": "2026-10", }, }, }) decision = decide({ "invoice_id": "INV-2026-0412", "client": "Orbit Print", "month": "2026-10", "total": 2350, "status": "unpaid", }) print(decision.value, decision.confidence)
Right invoice, but no totalanswer partly · confidence 0.998
Context
- app
- Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.
- today
- Monday 2 November 2026
- sub_question
- What is the invoice number and total of Orbit Print's October 2026 invoice?
- call
- tool: find_invoice description: Finds the studio's invoice for a client and month. arguments: {"client":"Orbit Print","month":"2026-10"}
Input
invoice_id: INV-2026-0412 client: Orbit Print month: 2026-10 status: unpaid
- Answer (JSON)
- value partly, confidence 0.998
- Top options
- partly 0.998
- yes 0.001
- no 0.001
The result is the right invoice but has no total, one of the two details asked, so the answer is
partly.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.agents.result-relevant.relevant", context={ "app": "Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.", "today": "Monday 2 November 2026", "sub_question": "What is the invoice number and total of Orbit Print's October 2026 invoice?", "call": { "tool": "find_invoice", "description": "Finds the studio's invoice for a client and month.", "arguments": { "client": "Orbit Print", "month": "2026-10", }, }, }) decision = decide({ "invoice_id": "INV-2026-0412", "client": "Orbit Print", "month": "2026-10", "status": "unpaid", }) print(decision.value, decision.confidence)
Tool answered with an erroranswer no · confidence 0.999
Context
- app
- Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.
- today
- Monday 2 November 2026
- sub_question
- What is the invoice number and total of Orbit Print's October 2026 invoice?
- call
- tool: find_invoice description: Finds the studio's invoice for a client and month. arguments: {"client":"Orbit Print","month":"2026-10"}
Input
status: error message: No invoice found for client "Orbit Print" in 2026-10.
- Answer (JSON)
- value no, confidence 0.999
- Top options
- no 0.999
- partly 0.001
- yes 0
The tool answered with an error instead of an invoice, so the answer is
no.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.agents.result-relevant.relevant", context={ "app": "Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.", "today": "Monday 2 November 2026", "sub_question": "What is the invoice number and total of Orbit Print's October 2026 invoice?", "call": { "tool": "find_invoice", "description": "Finds the studio's invoice for a client and month.", "arguments": { "client": "Orbit Print", "month": "2026-10", }, }, }) decision = decide({ "status": "error", "message": "No invoice found for client \"Orbit Print\" in 2026-10.", }) print(decision.value, decision.confidence)
First page of a list the question needs wholeanswer partly · confidence 0.963
Context
- app
- Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.
- today
- Monday 2 November 2026
- sub_question
- Which of the studio's invoices issued in October 2026 are still unpaid? List all of them.
- call
- tool: list_invoices description: Lists the studio's invoices for a month, 3 per page. arguments: {"month":"2026-10","status":"unpaid","page":1}
Input
invoices: [{"invoice_id":"INV-2026-0412","client":"Orbit Print","total":2350},{"invoice_id":"INV-2026-0415","client":"Moss & Gale","total":880},{"invoice_id":"INV-2026-0419","client":"Harbour Lane Cafe","total":640}] page: 1 pages: 2
- Answer (JSON)
- value partly, confidence 0.963
- Top options
- partly 0.963
- yes 0.022
- no 0.015
The question needs every unpaid invoice and only the first of two pages came back, so the answer is
partly.import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.agents.result-relevant.relevant", context={ "app": "Pinecrest Books, a bookkeeping assistant for a small design studio. It keeps the studio's invoices, clients and supplier bills, emails invoices and pays bills on request.", "today": "Monday 2 November 2026", "sub_question": "Which of the studio's invoices issued in October 2026 are still unpaid? List all of them.", "call": { "tool": "list_invoices", "description": "Lists the studio's invoices for a month, 3 per page.", "arguments": { "month": "2026-10", "status": "unpaid", "page": 1, }, }, }) decision = decide({ "invoices": [ { "invoice_id": "INV-2026-0412", "client": "Orbit Print", "total": 2350, }, { "invoice_id": "INV-2026-0415", "client": "Moss & Gale", "total": 880, }, { "invoice_id": "INV-2026-0419", "client": "Harbour Lane Cafe", "total": 640, }, ], "page": 1, "pages": 2, }) print(decision.value, decision.confidence)
Use when
- Your agent needs to check a search or tool result before proceeding.
Avoid when
- You need to assess completion of the entire task; use Agent task completion.
Context and input
| Field | Kind | Description |
|---|---|---|
result | input | The tool's result, as returned. |
app | context | The application the agent works in: its name and a description of what it does and for whom. |
today | context | Today's date, written out (for example 'Wednesday 23 June 2027'). |
sub_question | context | The agent's current sub-question. |
call | context | The tool call made: an object with tool, its description and arguments. |
Functions
| Key | Interface | Instructions | Labels |
|---|---|---|---|
| relevant | classify | Does the tool result answer the agent's current sub-question? Check the record (id, name), the period and every detail the sub-question asks for. yes: the result gives every asked detail for the right record or period (other records or extra details next to it do not matter). partly: the result is about the right record or set and gives at least one of the asked details but not all of them, or only part of a list the sub-question needs in full was returned (more remain). no: the result is about a different record or a different period, gives none of the asked details (only other details, even for the right record), is empty, or is an error. | no, partly, yes |
How it was measured
Test accuracy: the share of test rows whose top answer was right.
- 98.6% on the in-house test (2,000 rows). in-house: same generator as the training data
- 98.5% on a second in-house test of unseen cases. in-house: same generator as the training data
When confident
From the calibration set: at or above the threshold confidence, how often the answer is right, and how often the adapter is that confident. Below it, escalate.
| Function | Right when confident | Confident on | Threshold | Calibration rows |
|---|---|---|---|---|
| relevant | 99.2% | 100.0% | 0.37 | 2,000 |
Base: Jeff v1.4. Adapter trained: 25 October 2026.
Base
jeff-base v1.4
Call it
jeff.agents.result-relevant.relevant, with placeholders for its input and context.
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.agents.result-relevant.relevant", context={ "app": "<app>", "today": "<today>", "sub_question": "<sub_question>", "call": "<call>", }) decision = decide("<result>") print(decision.value, decision.confidence)
Download
Weights: mstrasser/jeff-adapter-result-relevant (v1.4)
Licence: Apache-2.0