Results

What each adapter adds

Each adapter is scored on its full held-out test set, which it never trained on, three ways: the untrained Qwen3.5-0.8B that Jeff is built from, Jeff v1.2 0.8B alone, and Jeff v1.2 0.8B with the adapter. Across the 9 adapters, mean accuracy goes from 29.5% untrained to 55.6% for Jeff alone and 91.5% with the adapter.

Accuracy is the share of test questions answered correctly. Calibration error (ECE) measures how far the stated confidence is from the real hit rate: 0.01 means the stated confidence is, on average, about 1 percentage point away from how often those answers are right. What calibration means, and why it matters.

Jeff + adapters against Qwen3.8-27B

On 8 of 9 tasks, Jeff with its adapter beats the 27B on its own; the 27B stays as a fallback you can tune per task, and for anything no adapter covers.

The adapters do the work: with the thresholds below, Jeff sends no query at all to Qwen3.8-27B on 8 of the 9 tasks. Both setups answered the same fixed random sample of each task's held-out rows on one Apple M4 Max, 128 GB; both models with MLX, one at a time.

• Qwen3.8-27B alone: the 27B answers every query. The model: Qwen3.8-27B, 8-bit MLX weights, prompted, step-by-step reasoning off; it uses 28.6 GB.
• Jeff + adapter first: Jeff + the task's adapter answers first; when its confidence is below the threshold, the 27B answers too and its answer is used (time = Jeff + 27B).
• The threshold: each task has its own, chosen on each task's own calibration rows: the lowest threshold (fastest) whose accuracy beats the 27B alone by at least 1 point, and then measured on the test rows. In plain words: Jeff only passes a query on to Qwen3.8-27B when that is needed to stay ahead.
• Timing: each task's fixed random sample of held-out rows; the same rows for both routes; times are the mean per query over the whole sample, prompt to answer.
• How certain: each gain has a 95% interval from a paired bootstrap over the test rows, 10,000 resamples per adapter (2,000 for the mean). “Tie” means the interval includes zero.
• Sample size: Qwen3.8-27B takes seconds per decision, so the comparison uses a fixed random sample of 300 rows per task (500 for emotion and legal-clauses). Jeff + adapter was also scored on each full test set (3,300 to 9,895 rows), and agrees with the sample to within 2.1 points on every adapter.

8 adapters

The headline is the mean over these 8 adapters, each at its own threshold; emotion is shown separately below.

Where the gain comes from

The same 8 adapters, the same rows, four setups. Against Qwen3.8-27B alone, the adapters on their own add 8.6 points of accuracy; routing on its own adds 0.5.

“Routing only” had its threshold picked on the test rows, because plain Jeff has no calibration rows, so its result is flattered.

ground is the only task that passes queries on to Qwen3.8-27B: when Jeff is less than 64% sure (threshold 0.64), which happened for 1.7% of its queries. On the test rows it scores 96.3% against Qwen3.8-27B's 96.7%: one question in 300, within noise, and the rule was fixed in advance on the calibration rows, not tuned on the test. So: about equal accuracy, 20× faster.

Every other task sends nothing to Qwen3.8-27B and is more accurate.

Left out of the averages

Emotion is left out of the averages: picking the single strongest of 27 emotions (or neutral) in short Reddit comments is hard even for people, and the human labels often disagree. Jeff + adapter scores 60.6% there against the 27B's 35.6%, at 42× the speed. Including it, the average across all adapters is 91.4% against 80.9%, so leaving it out makes the gain smaller, not larger.

One shared threshold, for comparison

The threshold trades speed for care. At 0 Jeff answers everything; at 1 Qwen3.8-27B answers everything. Here the same threshold is applied to every task at once, to show the whole dial; the dot marks 0.51, the single threshold used before each task got its own.

The base models

Jeff itself, before any adapter, on its general benchmark of 4,599 questions. The adapters on this site are trained on the 0.8B model.

Serving speed and memory

One Jeff server on an NVIDIA RTX PRO 6000: the base model loaded once, adapters beside it, one decision per request. With the default serving (LoRA in bfloat16), a decision with an adapter takes a median of 31.2 ms, against 25.9 ms for the base alone. With all nine adapters loaded and a different one on every request, it is 30.0 ms: switching costs nothing measurable. Merged mode folds one adapter into the weights and runs at the base's speed (25.7 ms), but then serves only that adapter.

675 requests per setting. For comparison, Qwen3.8-27B uses 28.6 GB on the Mac; Jeff's memory on the Mac is not measured yet.

Not measured yet

Nothing on this site is estimated. These are still to come:

• Jeff's memory on the Mac (measured on the RTX only)
• end-to-end ticket times (not run yet)

Where the numbers came from

• Adapters and serving: jeff-finetunes/adapters/BASELINE.md
• Against Qwen3.8-27B: jeff-reference-app results/cascade.json
• Jeff v1.2 0.8B: jev/runs/eval/0.8b-20260929-2258-final-calibrated.json
• Jeff v1.2 2B: jev/runs/eval/2b-20260930-2347-final-calibrated.json

Collected into one file on 2026-10-01T11:56:57+00:00.