Worked examples
CI failures routed to the right person
The problem
An open-source library runs its tests on every pull request. When a run fails, someone has to read the log and decide: is this a real bug in the change, a flaky test, a broken dependency or a problem with the CI machines? Then the right team has to hear about it. A flaky test that gets a bug report wastes a maintainer's morning, and a real regression that gets a rerun ships.
How Jeff helps
Two engineering functions do the sorting, each in one fast call:
- CI failure triage reads the failed run's log, the change and the facts the CI system records, and names the kind of failure.
- Issue routing reads the issue filed for a real failure against the project's components and open issues, and says what kind of issue it is and which component owns it.
Your CI bot then reruns, files an issue or pings the owning team.
The solution
Each step below is a live call; the answer under it is the one the API returned. Choose your language and whether to run it on your machine (free) or through the API.
A test fails after a code change
A pull request rewrites the table layout, and a table test fails on both attempts, while it passed on every recent run of the main branch:
Context
- project
- quillmark, an open-source Python library that turns Markdown into print-ready PDF files.
- ci_run
- job: tests (python 3.13) trigger: pull request 512 commit: 9e41c2b date: 4 November 2026
- change
- title: Rewrite the table layout to measure columns once description: Replaces the column-width loop in quillmark/layout/tables.py with a single measuring pass.
- ci_facts
- attempts_on_this_commit: 2; both failed with the same error same_test_on_main: passed in each of the last 50 runs other_jobs_in_this_run: lint passed; docs passed dependency_versions: unchanged since the last passing run
Input
tests/test_tables.py::test_wide_table_wraps FAILED E AssertionError: column 3 width 212.0 exceeds page width 180.0 ========== 1 failed, 388 passed in 52.3s ==========
- Answer (JSON)
- value real_regression, confidence 0.999
- Top options
- real_regression 0.999
- dependency_change 0
- environment_or_infra 0
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.engineering.ci-failure.failure", context={ "project": "quillmark, an open-source Python library that turns Markdown into print-ready PDF files.", "ci_run": { "job": "tests (python 3.13)", "trigger": "pull request 512", "commit": "9e41c2b", "date": "4 November 2026", }, "change": { "title": "Rewrite the table layout to measure columns once", "description": "Replaces the column-width loop in quillmark/layout/tables.py with a single measuring pass.", }, "ci_facts": { "attempts_on_this_commit": "2; both failed with the same error", "same_test_on_main": "passed in each of the last 50 runs", "other_jobs_in_this_run": "lint passed; docs passed", "dependency_versions": "unchanged since the last passing run", }, }) decision = decide("tests/test_tables.py::test_wide_table_wraps FAILED\nE AssertionError: column 3 width 212.0 exceeds page width 180.0\n========== 1 failed, 388 passed in 52.3s ==========") print(decision.value, decision.confidence)
real_regression: the change broke the test. The bot files an issue instead of rerunning.
A test fails once and passes on rerun
A documentation-only pull request fails a font cache test that also fails now and then on the main branch:
Context
- project
- quillmark, an open-source Python library that turns Markdown into print-ready PDF files.
- ci_run
- job: tests (python 3.12) trigger: pull request 514 commit: 1c07f3e date: 4 November 2026
- change
- title: Fix a broken link in the contributing guide description: Documentation only.
- ci_facts
- attempts_on_this_commit: 2; the first failed, the rerun passed same_test_on_main: failed 4 times in the last 40 runs, with no code change in between other_jobs_in_this_run: lint passed; docs passed dependency_versions: unchanged since the last passing run
Input
tests/test_fonts.py::test_font_cache_concurrent FAILED E TimeoutError: font cache lock not released after 5.0 s ========== 1 failed, 388 passed in 49.1s ==========
- Answer (JSON)
- value flaky_test, confidence 1
- Top options
- flaky_test 1
- dependency_change 0
- environment_or_infra 0
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.engineering.ci-failure.failure", context={ "project": "quillmark, an open-source Python library that turns Markdown into print-ready PDF files.", "ci_run": { "job": "tests (python 3.12)", "trigger": "pull request 514", "commit": "1c07f3e", "date": "4 November 2026", }, "change": { "title": "Fix a broken link in the contributing guide", "description": "Documentation only.", }, "ci_facts": { "attempts_on_this_commit": "2; the first failed, the rerun passed", "same_test_on_main": "failed 4 times in the last 40 runs, with no code change in between", "other_jobs_in_this_run": "lint passed; docs passed", "dependency_versions": "unchanged since the last passing run", }, }) decision = decide("tests/test_fonts.py::test_font_cache_concurrent FAILED\nE TimeoutError: font cache lock not released after 5.0 s\n========== 1 failed, 388 passed in 49.1s ==========") print(decision.value, decision.confidence)
flaky_test: the bot reruns it and adds the failure to the flaky test's existing issue, without bothering the
author.
Route the regression's issue
The issue filed for the regression:
Context
- project
- quillmark is an open-source Python library that turns Markdown into print-ready PDF files. Components: - parser (team syntax): the Markdown reader; paths quillmark/parser/ - layout (team layout): pages, tables and line breaking; paths quillmark/layout/ - fonts (team type): font loading and the font cache; paths quillmark/fonts/ - docs (team docs): the documentation site; paths docs/
- open_issues
- - #498 [fonts] Font cache test times out now and then - #503 [parser] Support footnotes inside tables
Input
type: issue number: 515 title: Wide tables overflow the page since pull request 512 body: After pull request 512 (measure table columns once), test_wide_table_wraps fails: a wide table's third column is 212 points wide on a 180-point page instead of wrapping. Main passed before the change. labels: []
- Answer (JSON)
- value bug, confidence 0.993
- Top options
- bug 0.993
- duplicate_candidate 0.006
- docs 0.001
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.engineering.issue-route.kind", context={ "project": "quillmark is an open-source Python library that turns Markdown into print-ready PDF files. Components:\n- parser (team syntax): the Markdown reader; paths quillmark/parser/\n- layout (team layout): pages, tables and line breaking; paths quillmark/layout/\n- fonts (team type): font loading and the font cache; paths quillmark/fonts/\n- docs (team docs): the documentation site; paths docs/", "open_issues": [ "#498 [fonts] Font cache test times out now and then", "#503 [parser] Support footnotes inside tables", ], }) decision = decide({ "type": "issue", "number": 515, "title": "Wide tables overflow the page since pull request 512", "body": "After pull request 512 (measure table columns once), test_wide_table_wraps fails: a wide table's third column is 212 points wide on a 180-point page instead of wrapping. Main passed before the change.", "labels": [], }) print(decision.value, decision.confidence)
bug: a fault in the code, not a question or a feature request.
Context
- project
- quillmark is an open-source Python library that turns Markdown into print-ready PDF files. Components: - parser (team syntax): the Markdown reader; paths quillmark/parser/ - layout (team layout): pages, tables and line breaking; paths quillmark/layout/ - fonts (team type): font loading and the font cache; paths quillmark/fonts/ - docs (team docs): the documentation site; paths docs/
- open_issues
- - #498 [fonts] Font cache test times out now and then - #503 [parser] Support footnotes inside tables
Input
type: issue number: 515 title: Wide tables overflow the page since pull request 512 body: After pull request 512 (measure table columns once), test_wide_table_wraps fails: a wide table's third column is 212 points wide on a 180-point page instead of wrapping. Main passed before the change. labels: []
Labels (the options)
- parser
- The Markdown reader (quillmark/parser/)
- layout
- Pages, tables and line breaking (quillmark/layout/)
- fonts
- Font loading and the font cache (quillmark/fonts/)
- docs
- The documentation site (docs/)
- Answer (JSON)
- value layout, confidence 0.995
- Top options
- layout 0.995
- parser 0.004
- fonts 0.001
import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client: decide = client.function("jeff.engineering.issue-route.owner", context={ "project": "quillmark is an open-source Python library that turns Markdown into print-ready PDF files. Components:\n- parser (team syntax): the Markdown reader; paths quillmark/parser/\n- layout (team layout): pages, tables and line breaking; paths quillmark/layout/\n- fonts (team type): font loading and the font cache; paths quillmark/fonts/\n- docs (team docs): the documentation site; paths docs/", "open_issues": [ "#498 [fonts] Font cache test times out now and then", "#503 [parser] Support footnotes inside tables", ], }, labels={ "parser": "The Markdown reader (quillmark/parser/)", "layout": "Pages, tables and line breaking (quillmark/layout/)", "fonts": "Font loading and the font cache (quillmark/fonts/)", "docs": "The documentation site (docs/)", }) decision = decide({ "type": "issue", "number": 515, "title": "Wide tables overflow the page since pull request 512", "body": "After pull request 512 (measure table columns once), test_wide_table_wraps fails: a wide table's third column is 212 points wide on a 180-point page instead of wrapping. Main passed before the change.", "labels": [], }) print(decision.value, decision.confidence)
layout: tables belong to the layout component, so the bot assigns the issue to its team.
What you get
- Every failed run sorted by cause and every issue routed to a team in two calls, with a confidence your bot can check before it acts. All four answers above were above their functions' thresholds.
- How the functions measure, from each adapter's page: CI failure triage 99.7% and issue routing 99.0% on their in-house tests (2,000 rows each, from the same generator as the training data). Their rows were kept only where a blind LLM check agreed with the label, so these scores are probably inflated, and neither has a public-benchmark figure. Check them on your own project's failures before letting the bot act alone.