jeff.engineering.ci-failurev1.4.0

Build failure triage

Classifies a failed build or test run using its log, code changes and run history. Distinguishes regressions, flaky tests, infrastructure issues, dependency changes and tests that need updating.

Open in playground

Example

Context

project: tidepool, an open-source Python library for reading and writing tide-gauge data files.
ci_run: job: tests (python 3.12) trigger: pull request 412 commit: 9f3c2e1 date: 2 November 2026
change: title: Speed up the CSV reader description: Reads rows in blocks of 4,096 instead of one at a time. No behaviour change intended.
ci_facts: attempts_on_this_commit: 2, both failed the same way same_test_on_main: passed in each of the last 30 runs other_jobs_in_this_run: lint passed; docs passed dependency_versions: unchanged since the last passing run duration: 3 min 40 s (usual 3 min 30 s)

Input

tests/test_csv.py::test_reads_last_row FAILED E AssertionError: expected 1,000 rows, got 904 E assert 904 == 1000 ========== 1 failed, 213 passed in 41.2s ==========

Answer (JSON)

value real_regression, confidence 0.996

Top options

real_regression 0.996
dependency_change 0.001
environment_or_infra 0.001

The test fails the same way twice on this commit and passes on the main branch, so the failure is a real_regression.

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.engineering.ci-failure.failure", context={        "project": "tidepool, an open-source Python library for reading and writing tide-gauge data files.",        "ci_run": {            "job": "tests (python 3.12)",            "trigger": "pull request 412",            "commit": "9f3c2e1",            "date": "2 November 2026",        },        "change": {            "title": "Speed up the CSV reader",            "description": "Reads rows in blocks of 4,096 instead of one at a time. No behaviour change intended.",        },        "ci_facts": {            "attempts_on_this_commit": "2, both failed the same way",            "same_test_on_main": "passed in each of the last 30 runs",            "other_jobs_in_this_run": "lint passed; docs passed",            "dependency_versions": "unchanged since the last passing run",            "duration": "3 min 40 s (usual 3 min 30 s)",        },    })     decision = decide("tests/test_csv.py::test_reads_last_row FAILED\nE   AssertionError: expected 1,000 rows, got 904\nE   assert 904 == 1000\n========== 1 failed, 213 passed in 41.2s ==========")     print(decision.value, decision.confidence)

More examples

Test that passed on a rerunflaky_test · 1

Context

project: tidepool, an open-source Python library for reading and writing tide-gauge data files.
ci_run: job: tests (python 3.12) trigger: pull request 415 commit: 4b7d0aa date: 3 November 2026
change: title: Fix a typo in the README description: Documentation only.
ci_facts: attempts_on_this_commit: 2; the first failed, the rerun passed same_test_on_main: failed 3 times in the last 30 runs, with no code change in between other_jobs_in_this_run: lint passed; docs passed dependency_versions: unchanged since the last passing run

Input

tests/test_download.py::test_parallel_fetch FAILED E AssertionError: expected 8 files, got 7 (one worker finished late) ========== 1 failed, 213 passed in 40.8s ==========

Answer (JSON)

value flaky_test, confidence 1

Top options

flaky_test 1
dependency_change 0
environment_or_infra 0

The rerun of the same commit passed and the test fails now and then on the main branch, so it is a flaky_test.

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.engineering.ci-failure.failure", context={        "project": "tidepool, an open-source Python library for reading and writing tide-gauge data files.",        "ci_run": {            "job": "tests (python 3.12)",            "trigger": "pull request 415",            "commit": "4b7d0aa",            "date": "3 November 2026",        },        "change": {            "title": "Fix a typo in the README",            "description": "Documentation only.",        },        "ci_facts": {            "attempts_on_this_commit": "2; the first failed, the rerun passed",            "same_test_on_main": "failed 3 times in the last 30 runs, with no code change in between",            "other_jobs_in_this_run": "lint passed; docs passed",            "dependency_versions": "unchanged since the last passing run",        },    })     decision = decide("tests/test_download.py::test_parallel_fetch FAILED\nE   AssertionError: expected 8 files, got 7 (one worker finished late)\n========== 1 failed, 213 passed in 40.8s ==========")     print(decision.value, decision.confidence)

Package mirror unreachableenvironment_or_infra · 1

Context

project: tidepool, an open-source Python library for reading and writing tide-gauge data files.
ci_run: job: tests (python 3.11) trigger: push to main commit: 7e21c90 date: 3 November 2026
change: title: Add a reader for the Kelso station format description: New reader module with its tests.
ci_facts: attempts_on_this_commit: 1 same_test_on_main: not reached other_jobs_in_this_run: docs failed with the same network error dependency_versions: could not be resolved

Input

Collecting numpy>=1.26 ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='pypi-mirror.ci.example', port=443): Max retries exceeded (Caused by NameResolutionError: Failed to resolve 'pypi-mirror.ci.example') Error: Process completed with exit code 1.

Answer (JSON)

value environment_or_infra, confidence 1

Top options

environment_or_infra 1
dependency_change 0
build_or_config 0

The run could not reach the package mirror and another job failed the same way, so the cause is environment_or_infra.

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.engineering.ci-failure.failure", context={        "project": "tidepool, an open-source Python library for reading and writing tide-gauge data files.",        "ci_run": {            "job": "tests (python 3.11)",            "trigger": "push to main",            "commit": "7e21c90",            "date": "3 November 2026",        },        "change": {            "title": "Add a reader for the Kelso station format",            "description": "New reader module with its tests.",        },        "ci_facts": {            "attempts_on_this_commit": 1,            "same_test_on_main": "not reached",            "other_jobs_in_this_run": "docs failed with the same network error",            "dependency_versions": "could not be resolved",        },    })     decision = decide("Collecting numpy>=1.26\nERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='pypi-mirror.ci.example', port=443): Max retries exceeded (Caused by NameResolutionError: Failed to resolve 'pypi-mirror.ci.example')\nError: Process completed with exit code 1.")     print(decision.value, decision.confidence)

Behaviour changed on purpose, test not updatedtest_needs_update · 0.996

Context

project: tidepool, an open-source Python library for reading and writing tide-gauge data files.
ci_run: job: tests (python 3.12) trigger: pull request 420 commit: c03e5b2 date: 3 November 2026
change: title: Report heights in metres instead of feet description: All readers now return heights in metres by default, as agreed in issue 401. Breaking change, noted in the changelog.
ci_facts: attempts_on_this_commit: 2, both failed the same way same_test_on_main: passed in each of the last 30 runs other_jobs_in_this_run: lint passed; docs passed dependency_versions: unchanged since the last passing run

Input

tests/test_units.py::test_default_height_unit FAILED E AssertionError: assert 'm' == 'ft' ========== 1 failed, 213 passed in 41.0s ==========

Answer (JSON)

value test_needs_update, confidence 0.996

Top options

test_needs_update 0.996
real_regression 0.003
timeout_or_resource 0

The change switches heights to metres on purpose and the test still expects feet, so the answer is test_needs_update.

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.engineering.ci-failure.failure", context={        "project": "tidepool, an open-source Python library for reading and writing tide-gauge data files.",        "ci_run": {            "job": "tests (python 3.12)",            "trigger": "pull request 420",            "commit": "c03e5b2",            "date": "3 November 2026",        },        "change": {            "title": "Report heights in metres instead of feet",            "description": "All readers now return heights in metres by default, as agreed in issue 401. Breaking change, noted in the changelog.",        },        "ci_facts": {            "attempts_on_this_commit": "2, both failed the same way",            "same_test_on_main": "passed in each of the last 30 runs",            "other_jobs_in_this_run": "lint passed; docs passed",            "dependency_versions": "unchanged since the last passing run",        },    })     decision = decide("tests/test_units.py::test_default_height_unit FAILED\nE   AssertionError: assert 'm' == 'ft'\n========== 1 failed, 213 passed in 41.0s ==========")     print(decision.value, decision.confidence)

Type check fails before the testsbuild_or_config · 0.995

Context

project: tidepool, an open-source Python library for reading and writing tide-gauge data files.
ci_run: job: typecheck and tests trigger: pull request 421 commit: 18aa4f9 date: 3 November 2026
change: title: Add a timezone option to the CSV reader description: New tz argument for read_csv.
ci_facts: attempts_on_this_commit: 1 same_test_on_main: not reached other_jobs_in_this_run: lint passed dependency_versions: unchanged since the last passing run

Input

Running mypy... tidepool/readers/csv.py:52: error: Argument "tz" to "localize" has incompatible type "str | None"; expected "str" [arg-type] Found 1 error in 1 file (checked 41 source files) Error: Process completed with exit code 1. Tests were not run.

Answer (JSON)

value build_or_config, confidence 0.995

Top options

build_or_config 0.995
dependency_change 0.001
test_needs_update 0.001

The type check failed before any test ran, so the failure is build_or_config.

Open in playground

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.engineering.ci-failure.failure", context={        "project": "tidepool, an open-source Python library for reading and writing tide-gauge data files.",        "ci_run": {            "job": "typecheck and tests",            "trigger": "pull request 421",            "commit": "18aa4f9",            "date": "3 November 2026",        },        "change": {            "title": "Add a timezone option to the CSV reader",            "description": "New tz argument for read_csv.",        },        "ci_facts": {            "attempts_on_this_commit": 1,            "same_test_on_main": "not reached",            "other_jobs_in_this_run": "lint passed",            "dependency_versions": "unchanged since the last passing run",        },    })     decision = decide("Running mypy...\ntidepool/readers/csv.py:52: error: Argument \"tz\" to \"localize\" has incompatible type \"str | None\"; expected \"str\"  [arg-type]\nFound 1 error in 1 file (checked 41 source files)\nError: Process completed with exit code 1. Tests were not run.")     print(decision.value, decision.confidence)

Use when

You need an initial classification of continuous integration failures before a maintainer investigates.

Avoid when

You need to fix the failure; this adapter only classifies it.

Context and input

Field	Kind	Description
log	input	The log of the failed run, as text.
project	context	The software project: its name and a description of what it is.
ci_run	context	The failed CI run: an object with job, trigger (for example a pull request), commit and date.
change	context	The code change being tested: an object with its title and description.
ci_facts	context	Facts the CI system records: attempts on this commit, the history of the same test or job on the main branch, other jobs in this run, dependency versions compared with the last passing run, and durations.

Functions

Key	Interface	Instructions	Labels
failure	classify	What kind of failure is this CI run? Use the log, the change (if there is one) and the CI facts (attempts on this commit, the history of the same test or job on the main branch, other jobs, dependency versions compared with the last passing run, durations). Apply the first rule that fits. build_or_config: the run failed before any test ran, in compiling, type checking, linting, packaging or the CI configuration itself (a bad pipeline file, a missing secret or variable, a wrong path), even if the change caused it. environment_or_infra: the error comes from the CI machine or an outside service (network or DNS errors, a package registry or mirror down, no disk space, a lost runner, a service container that did not start), not from the code; other jobs often fail the same way. flaky_test: the test fails only some of the time on the same code: a rerun of the same commit passed, or the test fails now and then on the main branch without code changes. timeout_or_resource: a test or job was stopped for exceeding a time limit or a memory limit (out of memory, killed, exit code 137) on every attempt. dependency_change: a third-party package resolves to a different version than in the last passing run (pulled in automatically, or upgraded by the change itself) and the error comes from that package. test_needs_update: the change says this behaviour was changed on purpose, and the failing test still expects the old behaviour. real_regression: the change broke behaviour it did not mean to change: the failure repeats on every attempt and the same test passes on the main branch. Warnings or retries in the log that did not stop the run do not decide the answer.	build_or_config, dependency_change, environment_or_infra, flaky_test, real_regression, test_needs_update, timeout_or_resource

How it was measured

Test accuracy: the share of test rows whose top answer was right.

99.7% on the in-house test (2,000 rows). in-house: same generator as the training data
99.9% on a second in-house test of unseen cases. in-house: same generator as the training data

When confident

From the calibration set: at or above the threshold confidence, how often the answer is right, and how often the adapter is that confident. Below it, escalate.

Function	Right when confident	Confident on	Threshold	Calibration rows
failure	99.6%	100.0%	0.60	1,500

Caveats

Training and test rows were kept only where a blind LLM check agreed with the label, so these scores are probably inflated.

Base: Jeff v1.4. Adapter trained: 25 October 2026.

Base

jeff-base v1.4

Call it

jeff.engineering.ci-failure.failure, with placeholders for its input and context.

import os import jeff with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:     decide = client.function("jeff.engineering.ci-failure.failure", context={        "project": "<project>",        "ci_run": "<ci_run>",        "change": "<change>",        "ci_facts": "<ci_facts>",    })     decision = decide("<log>")     print(decision.value, decision.confidence)

Download

Weights: mstrasser/jeff-adapter-ci-failure (v1.4)

Licence: Apache-2.0