Skip to content
JeffHub
SafetyOfficialReproducedVersion 0.1.0

spamSpam and phishing in SMS and email

Says whether a text message or email is spam or phishing, and whether it is legitimate, spam or a phishing scam.

Data: Public SMS and email spam and phishing collections

Trained on Jeff v1.2. Will be retrained on v1.3. Roadmap

Use it when

  • You receive text messages or emails and want a spam or phishing probability for each.
  • You want to tell phishing (a scam after details, passwords or money) apart from ordinary advertising spam.
  • You set your own threshold on the yes probability, rather than taking a hard yes or no.

Not a good fit when

  • You need an email's headers, links or attachments judged. Only the subject and readable body were used, and bodies over 6,000 characters were cut.
  • Your messages are not in English. The data sets are English.
  • You need to catch the newest scams. Much of the data is old (SMS from 2011 and 2022, most email from 2002 to 2005, phishing email up to 2025), so recent tricks may be missed.
  • You block messages with no human check. Treat the answer as a signal, not a verdict.

Request format

The state is an object with these fields, in this order. Only message changes from request to request, so it comes last and the rest can be prepared in advance.

State fieldChanges per requestWhat goes in it
channelNoWhere the message came from: "sms" or "email", as in training.
messageYesThe text of the message as received. For an email, its subject and readable body (no other headers, no attachments).
QuestionTypeWhat it decides
is_spamYes or noWhether the message is spam (unwanted bulk or advertising) or phishing (a scam after personal details or money), rather than a normal message.
message_typeChoiceWhat kind of message it is.

Options: Three options: legitimate, spam and phishing, each with a one-line description (SPAM_CHOICE in descriptions.py in the source).

  • Ask both questions in one request if you need both; they are answered together.
  • For is_spam, give the false and true descriptions below, in that order; the adapter was trained with them.
  • The yes/no question was trained on every message. The three-way question was trained only where the source separates phishing from spam, the SMS phishing set and the SpamAssassin and Nazario emails.
  • Use the instructions below word for word; the adapter was trained mostly on them.

General rules for every request are in the request format guide.

Example

The same request three ways. It assumes a Jeff server on your machine with this adapter loaded (see Install).

from jeff import Client
from jeff.client import yes_no_question, choice_question

jeff = Client("http://localhost:8765", model="spam")

state = {
    "channel": "sms",
    "message": "Your parcel could not be delivered. Pay the 1.99 redelivery fee within 24 hours at parcel-redeliver-help.example to avoid return.",
}

answers = jeff.ask(state, {
    "is_spam": yes_no_question("Is this message spam or phishing? Answer yes if it is unwanted bulk or advertising, or a scam trying to get personal details or money; answer no if it is a normal message.", yes="The message is spam (unwanted bulk or advertising) or phishing (a scam trying to get personal details, passwords or money).", no="The message is a normal message: not unwanted bulk or advertising, and not a scam."),
    "message_type": choice_question(
        {
            "legitimate": "A normal message from a person or a genuine organisation, not spam and not phishing.",
            "spam": "Unwanted bulk or advertising message (for example prizes, offers, premium-rate services), but not trying to steal personal or account details.",
            "phishing": "A scam message that tries to trick the reader into giving personal details, passwords or money, often by pretending to be a bank, company or authority and asking them to click a link or call a number.",
        },
        "What kind of message is this: a legitimate message, spam (unwanted bulk or advertising), or phishing (a scam trying to get personal details, passwords or money)?",
    ),
})
print("is_spam", answers.yes_no("is_spam"))
print("message_type", answers.choice("message_type").key)

Response

{
  "model": "spam",
  "answers": {
    "is_spam": {
      "type": "noul",
      "noul": 0.9994498949457411
    },
    "message_type": {
      "type": "choice",
      "probabilities": {
        "legitimate": 0.0011454269371024618,
        "spam": 0.10794330903524028,
        "phishing": 0.8909112640276573
      },
      "choice": "phishing",
      "confidence": 0.836366896041486
    }
  },
  "usage": {
    "input_tokens": 443,
    "output_tokens": 0,
    "orders": 1
  }
}

Results

On this adapter's held-out test set, never trained on. Measured 2026-10-01. All adapters

  • spam3,603 test rows
    Qwen3.5-0.8B untrained
    59.4% · 0.060
    Jeff v1.2 0.8B alone
    72.3% · 0.109
    98.4% · 0.008

Each cell: accuracy · calibration error (ECE; lower is better, 0 is perfect).

How sure is it, and is it right?

Jeff gives every answer a probability. Each dot is a group of test rows with similar confidence: across, how sure the model said it was; up, how often it was right. Dots on the diagonal mean the stated confidence can be taken at face value.

When this adapter says it is about 99.8% sure, it is right about 99.4% of the time (3,459 test rows).

Jeff v1.2 0.8B aloneJeff v1.2 0.8B + adapterperfectly calibrated
0%0%25%25%50%50%75%75%100%100%Stated confidenceRight answersJeff v1.2 0.8B alone: 21 rows stated 38% on average and were right 33.3% of the timeJeff v1.2 0.8B alone: 70 rows stated 44% on average and were right 38.6% of the timeJeff v1.2 0.8B alone: 188 rows stated 50% on average and were right 46.3% of the timeJeff v1.2 0.8B alone: 210 rows stated 57% on average and were right 53.8% of the timeJeff v1.2 0.8B alone: 211 rows stated 64% on average and were right 61.6% of the timeJeff v1.2 0.8B alone: 230 rows stated 70% on average and were right 63.5% of the timeJeff v1.2 0.8B alone: 255 rows stated 77% on average and were right 61.6% of the timeJeff v1.2 0.8B alone: 337 rows stated 84% on average and were right 61.1% of the timeJeff v1.2 0.8B alone: 594 rows stated 90% on average and were right 65.7% of the timeJeff v1.2 0.8B alone: 1487 rows stated 96% on average and were right 90.2% of the timeJeff v1.2 0.8B + adapter: 1 rows stated 46% on average and were right 100.0% of the timeJeff v1.2 0.8B + adapter: 12 rows stated 52% on average and were right 41.7% of the timeJeff v1.2 0.8B + adapter: 17 rows stated 57% on average and were right 70.6% of the timeJeff v1.2 0.8B + adapter: 17 rows stated 65% on average and were right 58.8% of the timeJeff v1.2 0.8B + adapter: 12 rows stated 70% on average and were right 91.7% of the timeJeff v1.2 0.8B + adapter: 21 rows stated 76% on average and were right 81.0% of the timeJeff v1.2 0.8B + adapter: 25 rows stated 83% on average and were right 88.0% of the timeJeff v1.2 0.8B + adapter: 39 rows stated 90% on average and were right 76.9% of the timeJeff v1.2 0.8B + adapter: 3459 rows stated 100% on average and were right 99.4% of the time

Point at a dot for its numbers. Bigger dots hold more rows.

Calibration scores
ModelCalibration error (ECE)Brier scoreLog loss
Qwen3.5-0.8B untrained0.0600.4930.756
Jeff v1.2 0.8B alone0.1090.4050.664
Jeff v1.2 0.8B + adapter0.0080.0250.052

All three: lower is better, 0 is perfect. Brier score and log loss also reward being right.

Where it gets things wrong

  • yes read as no: 18 rows
  • no read as yes: 13 rows
  • spam read as phishing: 7 rows
  • phishing read as spam: 6 rows
  • spam read as legitimate: 5 rows

The five commonest mistakes with the adapter, out of 3,603 test rows.

Accuracy per right answer
Right answerTest rowsAccuracy with the adapter
no1,52299.1%
legitimate1,03399.5%
yes66297.3%
phishing27797.1%
spam10989.0%

By kind of question

  • is_spam: Whether the message is spam (unwanted bulk or advertising) or phishing (a scam after personal details or money), rather than a normal message.2,184 rows · 98.6%
  • message_type: What kind of message it is.1,419 rows · 98.2%

Accuracy with the adapter on each kind of question.

By source

  • choice: legitimate / phishing / spam (email)144 test rows
    Qwen3.5-0.8B untrained
    57.6% · 0.139
    Jeff v1.2 0.8B alone
    69.4% · 0.113
    97.2% · 0.029
  • choice: legitimate / phishing / spam (sms)82 test rows
    Qwen3.5-0.8B untrained
    81.7% · 0.357
    Jeff v1.2 0.8B alone
    80.5% · 0.150
    95.1% · 0.031
  • choice: legitimate / spam / phishing (email)149 test rows
    Qwen3.5-0.8B untrained
    29.5% · 0.256
    Jeff v1.2 0.8B alone
    64.4% · 0.158
    100.0% · 0.008
  • choice: legitimate / spam / phishing (sms)108 test rows
    Qwen3.5-0.8B untrained
    21.3% · 0.210
    Jeff v1.2 0.8B alone
    88.9% · 0.181
    97.2% · 0.023
  • choice: phishing / legitimate / spam (email)134 test rows
    Qwen3.5-0.8B untrained
    69.4% · 0.149
    Jeff v1.2 0.8B alone
    77.6% · 0.104
    99.3% · 0.008
  • choice: phishing / legitimate / spam (sms)102 test rows
    Qwen3.5-0.8B untrained
    87.3% · 0.291
    Jeff v1.2 0.8B alone
    79.4% · 0.170
    99.0% · 0.025
  • choice: phishing / spam / legitimate (email)123 test rows
    Qwen3.5-0.8B untrained
    65.0% · 0.183
    Jeff v1.2 0.8B alone
    69.1% · 0.104
    100.0% · 0.011
  • choice: phishing / spam / legitimate (sms)93 test rows
    Qwen3.5-0.8B untrained
    79.6% · 0.277
    Jeff v1.2 0.8B alone
    84.9% · 0.196
    98.9% · 0.016
  • choice: spam / legitimate / phishing (email)157 test rows
    Qwen3.5-0.8B untrained
    70.1% · 0.103
    Jeff v1.2 0.8B alone
    65.0% · 0.120
    97.5% · 0.018
  • choice: spam / legitimate / phishing (sms)110 test rows
    Qwen3.5-0.8B untrained
    83.6% · 0.226
    Jeff v1.2 0.8B alone
    75.5% · 0.141
    98.2% · 0.043
  • choice: spam / phishing / legitimate (email)139 test rows
    Qwen3.5-0.8B untrained
    71.9% · 0.100
    Jeff v1.2 0.8B alone
    69.8% · 0.068
    98.6% · 0.018
  • choice: spam / phishing / legitimate (sms)78 test rows
    Qwen3.5-0.8B untrained
    84.6% · 0.240
    Jeff v1.2 0.8B alone
    78.2% · 0.157
    96.2% · 0.026
  • yes/no question (email)1,592 test rows
    Qwen3.5-0.8B untrained
    65.1% · 0.091
    Jeff v1.2 0.8B alone
    66.6% · 0.234
    98.5% · 0.008
  • yes/no question (sms)592 test rows
    Qwen3.5-0.8B untrained
    30.9% · 0.271
    Jeff v1.2 0.8B alone
    83.6% · 0.071
    98.8% · 0.010

Each cell: accuracy · calibration error (ECE; lower is better, 0 is perfect).

Against Qwen3.8-27B

More accurate, 37× faster than Qwen3.8-27B alone.

Gain over Qwen3.8-27B alone: +10.7 points [+7.0, +14.7] (95% interval).

On 300 sampled test rows on an Apple M4 Max, 128 GB: Qwen3.8-27B alone was right 88.0% of the time at 2.67 s per query on average; with Jeff and this adapter answering first, it was right 98.7% at 73 ms.

This task's threshold is 0: Jeff stays ahead of Qwen3.8-27B without passing anything on, so it answered every query itself. Each task's threshold is the fastest one that still beats Qwen3.8-27B alone by at least 1 point on that task's calibration rows.

Time per query and prompt length for both routes
RouteMeanMedian95th percentileTypical prompt
Qwen3.8-27B alone2.67 s2.25 s5.02 s319 tokens
Jeff + adapter, its own answer73 ms61 ms126 ms286 tokens

Time per query, prompt to answer. Jeff's row is its own answer, before any hand-off. Typical prompt: the median prompt length in tokens.

This task's threshold 0.00: accuracy 98.7%, 36.6× faster, 0.0% sent on to Qwen3.8-27B.

Accuracy98.7%
85.0%92.5%100.0%0.000.250.500.751.00
Speed-up36.6×
0.0×20.0×40.0×0.000.250.500.751.00
Sent on to Qwen3.8-27B0.0%
0.0%50.0%100.0%0.000.250.500.751.00

Below the threshold, Jeff passes the query on to Qwen3.8-27B. Horizontal axis: the threshold, from 0 (Jeff answers everything) to 1 (Qwen3.8-27B answers everything). The dot and the vertical line mark the this task's threshold. On this task's 300 sampled rows; speed-up is Qwen3.8-27B's mean time divided by the route's mean time. Point at a chart to read any threshold.

All tasks, and how this was measured

How it was trained

Training rows
30,704
Steps
480
Training time
32 min
Size as saved
41.5 MB

One pass over the data (1 epoch) on one NVIDIA RTX PRO 6000. Run 0.8b-spam-20260930-1343.

Source: jeff-finetunes/adapters/BASELINE.md

Data card

Reproduced. Re-measured by the maintainers on a fixed 300-row sample of the test set, on a different machine and software (Apple M4 Max, MLX), within about 1.5 points of the full-test-set result. What the levels mean

How the test set was held out
10% of the text messages and 10% of the emails, held out by a stable hash of the text (no source set has an official test split); never trained on. Scored on both questions.
Training data
Built from public data sets, listed under Data and licence.
The source data sets are public (listed under Data and licence). A script to rebuild our rows from them will follow.

Data and licence

The adapter is released under Apache-2.0. It was trained on:

Changelog

  1. 0.1.0 · 2026-09-30Trained on Jeff v1.2 (LoRA rank 16, one epoch). Results on the Results page. Published on Hugging Face as v1.2, with its test and calibration sets. Three of the email sources give no licence.

Comments

Comments open when JeffHub launches. They will live in the registry repository's GitHub Discussions, one thread per adapter; you sign in with GitHub, and JeffHub stores no accounts.