spamSpam and phishing in SMS and email
Says whether a text message or email is spam or phishing, and whether it is legitimate, spam or a phishing scam.
Data: Public SMS and email spam and phishing collections
Trained on Jeff v1.2. Will be retrained on v1.3. Roadmap
Use it when
- You receive text messages or emails and want a spam or phishing probability for each.
- You want to tell phishing (a scam after details, passwords or money) apart from ordinary advertising spam.
- You set your own threshold on the yes probability, rather than taking a hard yes or no.
Not a good fit when
- You need an email's headers, links or attachments judged. Only the subject and readable body were used, and bodies over 6,000 characters were cut.
- Your messages are not in English. The data sets are English.
- You need to catch the newest scams. Much of the data is old (SMS from 2011 and 2022, most email from 2002 to 2005, phishing email up to 2025), so recent tricks may be missed.
- You block messages with no human check. Treat the answer as a signal, not a verdict.
Request format
The state is an object with these fields, in this order. Only message changes from request to request, so it comes last and the rest can be prepared in advance.
| State field | Changes per request | What goes in it |
|---|---|---|
channel | No | Where the message came from: "sms" or "email", as in training. |
message | Yes | The text of the message as received. For an email, its subject and readable body (no other headers, no attachments). |
| Question | Type | What it decides |
|---|---|---|
is_spam | Yes or no | Whether the message is spam (unwanted bulk or advertising) or phishing (a scam after personal details or money), rather than a normal message. |
message_type | Choice | What kind of message it is. Options: Three options: legitimate, spam and phishing, each with a one-line description (SPAM_CHOICE in descriptions.py in the source). |
- Ask both questions in one request if you need both; they are answered together.
- For is_spam, give the false and true descriptions below, in that order; the adapter was trained with them.
- The yes/no question was trained on every message. The three-way question was trained only where the source separates phishing from spam, the SMS phishing set and the SpamAssassin and Nazario emails.
- Use the instructions below word for word; the adapter was trained mostly on them.
General rules for every request are in the request format guide.
Example
The same request three ways. It assumes a Jeff server on your machine with this adapter loaded (see Install).
from jeff import Client
from jeff.client import yes_no_question, choice_question
jeff = Client("http://localhost:8765", model="spam")
state = {
"channel": "sms",
"message": "Your parcel could not be delivered. Pay the 1.99 redelivery fee within 24 hours at parcel-redeliver-help.example to avoid return.",
}
answers = jeff.ask(state, {
"is_spam": yes_no_question("Is this message spam or phishing? Answer yes if it is unwanted bulk or advertising, or a scam trying to get personal details or money; answer no if it is a normal message.", yes="The message is spam (unwanted bulk or advertising) or phishing (a scam trying to get personal details, passwords or money).", no="The message is a normal message: not unwanted bulk or advertising, and not a scam."),
"message_type": choice_question(
{
"legitimate": "A normal message from a person or a genuine organisation, not spam and not phishing.",
"spam": "Unwanted bulk or advertising message (for example prizes, offers, premium-rate services), but not trying to steal personal or account details.",
"phishing": "A scam message that tries to trick the reader into giving personal details, passwords or money, often by pretending to be a bank, company or authority and asking them to click a link or call a number.",
},
"What kind of message is this: a legitimate message, spam (unwanted bulk or advertising), or phishing (a scam trying to get personal details, passwords or money)?",
),
})
print("is_spam", answers.yes_no("is_spam"))
print("message_type", answers.choice("message_type").key)import { Client, yesNoQuestion, choiceQuestion } from '@jeff/client';
const jeff = new Client({ url: 'http://localhost:8765', model: 'spam' });
const state = {
channel: 'sms',
message: 'Your parcel could not be delivered. Pay the 1.99 redelivery fee within 24 hours at parcel-redeliver-help.example to avoid return.',
};
const answers = await jeff.ask(state, {
is_spam: yesNoQuestion('Is this message spam or phishing? Answer yes if it is unwanted bulk or advertising, or a scam trying to get personal details or money; answer no if it is a normal message.', { yes: 'The message is spam (unwanted bulk or advertising) or phishing (a scam trying to get personal details, passwords or money).', no: 'The message is a normal message: not unwanted bulk or advertising, and not a scam.' }),
message_type: choiceQuestion(
{
legitimate: 'A normal message from a person or a genuine organisation, not spam and not phishing.',
spam: 'Unwanted bulk or advertising message (for example prizes, offers, premium-rate services), but not trying to steal personal or account details.',
phishing: 'A scam message that tries to trick the reader into giving personal details, passwords or money, often by pretending to be a bank, company or authority and asking them to click a link or call a number.',
},
'What kind of message is this: a legitimate message, spam (unwanted bulk or advertising), or phishing (a scam trying to get personal details, passwords or money)?',
),
});
console.log('is_spam', answers.is_spam);
console.log('message_type', answers.message_type.key);curl -s http://localhost:8765/v1/systemone \
-H 'content-type: application/json' \
-d '{
"model": "spam",
"state": {
"channel": "sms",
"message": "Your parcel could not be delivered. Pay the 1.99 redelivery fee within 24 hours at parcel-redeliver-help.example to avoid return."
},
"questions": {
"is_spam": {
"type": "noul",
"instructions": "Is this message spam or phishing? Answer yes if it is unwanted bulk or advertising, or a scam trying to get personal details or money; answer no if it is a normal message.",
"criteria": {
"true": "The message is spam (unwanted bulk or advertising) or phishing (a scam trying to get personal details, passwords or money).",
"false": "The message is a normal message: not unwanted bulk or advertising, and not a scam."
}
},
"message_type": {
"type": "choice",
"instructions": "What kind of message is this: a legitimate message, spam (unwanted bulk or advertising), or phishing (a scam trying to get personal details, passwords or money)?",
"criteria": {
"legitimate": "A normal message from a person or a genuine organisation, not spam and not phishing.",
"spam": "Unwanted bulk or advertising message (for example prizes, offers, premium-rate services), but not trying to steal personal or account details.",
"phishing": "A scam message that tries to trick the reader into giving personal details, passwords or money, often by pretending to be a bank, company or authority and asking them to click a link or call a number."
}
}
}
}'Response
{
"model": "spam",
"answers": {
"is_spam": {
"type": "noul",
"noul": 0.9994498949457411
},
"message_type": {
"type": "choice",
"probabilities": {
"legitimate": 0.0011454269371024618,
"spam": 0.10794330903524028,
"phishing": 0.8909112640276573
},
"choice": "phishing",
"confidence": 0.836366896041486
}
},
"usage": {
"input_tokens": 443,
"output_tokens": 0,
"orders": 1
}
}Results
On this adapter's held-out test set, never trained on. Measured 2026-10-01. All adapters
| Test set | Test rows | Qwen3.5-0.8B untrained | Jeff v1.2 0.8B alone | Jeff v1.2 0.8B + adapter |
|---|---|---|---|---|
spam | 3,603 | 59.4% · 0.060 | 72.3% · 0.109 | 98.4% · 0.008 |
spam3,603 test rows- Qwen3.5-0.8B untrained
- 59.4% · 0.060
- Jeff v1.2 0.8B alone
- 72.3% · 0.109
- Jeff v1.2 0.8B + adapter
- 98.4% · 0.008
Each cell: accuracy · calibration error (ECE; lower is better, 0 is perfect).
How sure is it, and is it right?
Jeff gives every answer a probability. Each dot is a group of test rows with similar confidence: across, how sure the model said it was; up, how often it was right. Dots on the diagonal mean the stated confidence can be taken at face value.
When this adapter says it is about 99.8% sure, it is right about 99.4% of the time (3,459 test rows).
Point at a dot for its numbers. Bigger dots hold more rows.
| Model | Calibration error (ECE) | Brier score | Log loss |
|---|---|---|---|
| Qwen3.5-0.8B untrained | 0.060 | 0.493 | 0.756 |
| Jeff v1.2 0.8B alone | 0.109 | 0.405 | 0.664 |
| Jeff v1.2 0.8B + adapter | 0.008 | 0.025 | 0.052 |
All three: lower is better, 0 is perfect. Brier score and log loss also reward being right.
Where it gets things wrong
- yes read as no: 18 rows
- no read as yes: 13 rows
spamread asphishing: 7 rowsphishingread asspam: 6 rowsspamread aslegitimate: 5 rows
The five commonest mistakes with the adapter, out of 3,603 test rows.
| Right answer | Test rows | Accuracy with the adapter |
|---|---|---|
| no | 1,522 | 99.1% |
legitimate | 1,033 | 99.5% |
| yes | 662 | 97.3% |
phishing | 277 | 97.1% |
spam | 109 | 89.0% |
By kind of question
is_spam: Whether the message is spam (unwanted bulk or advertising) or phishing (a scam after personal details or money), rather than a normal message.2,184 rows · 98.6%message_type: What kind of message it is.1,419 rows · 98.2%
Accuracy with the adapter on each kind of question.
By source
| Source | Test rows | Qwen3.5-0.8B untrained | Jeff v1.2 0.8B alone | Jeff v1.2 0.8B + adapter |
|---|---|---|---|---|
choice: legitimate / phishing / spam (email) | 144 | 57.6% · 0.139 | 69.4% · 0.113 | 97.2% · 0.029 |
choice: legitimate / phishing / spam (sms) | 82 | 81.7% · 0.357 | 80.5% · 0.150 | 95.1% · 0.031 |
choice: legitimate / spam / phishing (email) | 149 | 29.5% · 0.256 | 64.4% · 0.158 | 100.0% · 0.008 |
choice: legitimate / spam / phishing (sms) | 108 | 21.3% · 0.210 | 88.9% · 0.181 | 97.2% · 0.023 |
choice: phishing / legitimate / spam (email) | 134 | 69.4% · 0.149 | 77.6% · 0.104 | 99.3% · 0.008 |
choice: phishing / legitimate / spam (sms) | 102 | 87.3% · 0.291 | 79.4% · 0.170 | 99.0% · 0.025 |
choice: phishing / spam / legitimate (email) | 123 | 65.0% · 0.183 | 69.1% · 0.104 | 100.0% · 0.011 |
choice: phishing / spam / legitimate (sms) | 93 | 79.6% · 0.277 | 84.9% · 0.196 | 98.9% · 0.016 |
choice: spam / legitimate / phishing (email) | 157 | 70.1% · 0.103 | 65.0% · 0.120 | 97.5% · 0.018 |
choice: spam / legitimate / phishing (sms) | 110 | 83.6% · 0.226 | 75.5% · 0.141 | 98.2% · 0.043 |
choice: spam / phishing / legitimate (email) | 139 | 71.9% · 0.100 | 69.8% · 0.068 | 98.6% · 0.018 |
choice: spam / phishing / legitimate (sms) | 78 | 84.6% · 0.240 | 78.2% · 0.157 | 96.2% · 0.026 |
yes/no question (email) | 1,592 | 65.1% · 0.091 | 66.6% · 0.234 | 98.5% · 0.008 |
yes/no question (sms) | 592 | 30.9% · 0.271 | 83.6% · 0.071 | 98.8% · 0.010 |
choice: legitimate / phishing / spam (email)144 test rows- Qwen3.5-0.8B untrained
- 57.6% · 0.139
- Jeff v1.2 0.8B alone
- 69.4% · 0.113
- Jeff v1.2 0.8B + adapter
- 97.2% · 0.029
choice: legitimate / phishing / spam (sms)82 test rows- Qwen3.5-0.8B untrained
- 81.7% · 0.357
- Jeff v1.2 0.8B alone
- 80.5% · 0.150
- Jeff v1.2 0.8B + adapter
- 95.1% · 0.031
choice: legitimate / spam / phishing (email)149 test rows- Qwen3.5-0.8B untrained
- 29.5% · 0.256
- Jeff v1.2 0.8B alone
- 64.4% · 0.158
- Jeff v1.2 0.8B + adapter
- 100.0% · 0.008
choice: legitimate / spam / phishing (sms)108 test rows- Qwen3.5-0.8B untrained
- 21.3% · 0.210
- Jeff v1.2 0.8B alone
- 88.9% · 0.181
- Jeff v1.2 0.8B + adapter
- 97.2% · 0.023
choice: phishing / legitimate / spam (email)134 test rows- Qwen3.5-0.8B untrained
- 69.4% · 0.149
- Jeff v1.2 0.8B alone
- 77.6% · 0.104
- Jeff v1.2 0.8B + adapter
- 99.3% · 0.008
choice: phishing / legitimate / spam (sms)102 test rows- Qwen3.5-0.8B untrained
- 87.3% · 0.291
- Jeff v1.2 0.8B alone
- 79.4% · 0.170
- Jeff v1.2 0.8B + adapter
- 99.0% · 0.025
choice: phishing / spam / legitimate (email)123 test rows- Qwen3.5-0.8B untrained
- 65.0% · 0.183
- Jeff v1.2 0.8B alone
- 69.1% · 0.104
- Jeff v1.2 0.8B + adapter
- 100.0% · 0.011
choice: phishing / spam / legitimate (sms)93 test rows- Qwen3.5-0.8B untrained
- 79.6% · 0.277
- Jeff v1.2 0.8B alone
- 84.9% · 0.196
- Jeff v1.2 0.8B + adapter
- 98.9% · 0.016
choice: spam / legitimate / phishing (email)157 test rows- Qwen3.5-0.8B untrained
- 70.1% · 0.103
- Jeff v1.2 0.8B alone
- 65.0% · 0.120
- Jeff v1.2 0.8B + adapter
- 97.5% · 0.018
choice: spam / legitimate / phishing (sms)110 test rows- Qwen3.5-0.8B untrained
- 83.6% · 0.226
- Jeff v1.2 0.8B alone
- 75.5% · 0.141
- Jeff v1.2 0.8B + adapter
- 98.2% · 0.043
choice: spam / phishing / legitimate (email)139 test rows- Qwen3.5-0.8B untrained
- 71.9% · 0.100
- Jeff v1.2 0.8B alone
- 69.8% · 0.068
- Jeff v1.2 0.8B + adapter
- 98.6% · 0.018
choice: spam / phishing / legitimate (sms)78 test rows- Qwen3.5-0.8B untrained
- 84.6% · 0.240
- Jeff v1.2 0.8B alone
- 78.2% · 0.157
- Jeff v1.2 0.8B + adapter
- 96.2% · 0.026
yes/no question (email)1,592 test rows- Qwen3.5-0.8B untrained
- 65.1% · 0.091
- Jeff v1.2 0.8B alone
- 66.6% · 0.234
- Jeff v1.2 0.8B + adapter
- 98.5% · 0.008
yes/no question (sms)592 test rows- Qwen3.5-0.8B untrained
- 30.9% · 0.271
- Jeff v1.2 0.8B alone
- 83.6% · 0.071
- Jeff v1.2 0.8B + adapter
- 98.8% · 0.010
Each cell: accuracy · calibration error (ECE; lower is better, 0 is perfect).
Against Qwen3.8-27B
More accurate, 37× faster than Qwen3.8-27B alone.
Gain over Qwen3.8-27B alone: +10.7 points [+7.0, +14.7] (95% interval).
On 300 sampled test rows on an Apple M4 Max, 128 GB: Qwen3.8-27B alone was right 88.0% of the time at 2.67 s per query on average; with Jeff and this adapter answering first, it was right 98.7% at 73 ms.
This task's threshold is 0: Jeff stays ahead of Qwen3.8-27B without passing anything on, so it answered every query itself. Each task's threshold is the fastest one that still beats Qwen3.8-27B alone by at least 1 point on that task's calibration rows.
| Route | Mean | Median | 95th percentile | Typical prompt |
|---|---|---|---|---|
| Qwen3.8-27B alone | 2.67 s | 2.25 s | 5.02 s | 319 tokens |
| Jeff + adapter, its own answer | 73 ms | 61 ms | 126 ms | 286 tokens |
Time per query, prompt to answer. Jeff's row is its own answer, before any hand-off. Typical prompt: the median prompt length in tokens.
This task's threshold 0.00: accuracy 98.7%, 36.6× faster, 0.0% sent on to Qwen3.8-27B.
Below the threshold, Jeff passes the query on to Qwen3.8-27B. Horizontal axis: the threshold, from 0 (Jeff answers everything) to 1 (Qwen3.8-27B answers everything). The dot and the vertical line mark the this task's threshold. On this task's 300 sampled rows; speed-up is Qwen3.8-27B's mean time divided by the route's mean time. Point at a chart to read any threshold.
How it was trained
- Training rows
- 30,704
- Steps
- 480
- Training time
- 32 min
- Size as saved
- 41.5 MB
One pass over the data (1 epoch) on one NVIDIA RTX PRO 6000. Run 0.8b-spam-20260930-1343.
Source: jeff-finetunes/adapters/BASELINE.md
Data card
Reproduced. Re-measured by the maintainers on a fixed 300-row sample of the test set, on a different machine and software (Apple M4 Max, MLX), within about 1.5 points of the full-test-set result. What the levels mean
- Test set, so anyone can check the numbers
- Calibration rows, the rows its threshold is chosen on
- QA report, sanitised: the data-quality checks run before training
- How the test set was held out
- 10% of the text messages and 10% of the emails, held out by a stable hash of the text (no source set has an official test split); never trained on. Scored on both questions.
- Training data
- Built from public data sets, listed under Data and licence.
- The source data sets are public (listed under Data and licence). A script to rebuild our rows from them will follow.
Data and licence
The adapter is released under Apache-2.0. It was trained on:
- UCI SMS Spam Collection (Almeida and Hidalgo 2011)Licence: CC-BY-4.0
Labels ham and spam.
- SMS Phishing Dataset for Machine Learning and Pattern Recognition (Mishra and Soni 2022), version 1Licence: CC-BY-4.0
Labels ham, spam and smishing (SMS phishing). Messages shared with the UCI set were merged by text.
- Phishing Email Dataset (zefang-liu, a copy of the Kaggle set "Phishing Email Detection")Licence: No licence given by the source
Tagged LGPL-3.0 by the uploader; the Kaggle original does not say where its emails come from. Labels safe and phishing.
- ealvaradob/phishing-dataset (emails only)Licence: No licence given by the source
Tagged Apache-2.0 by the compiler; its emails are the same Kaggle set as above, so it adds only a handful of messages.
- SpamAssassin public mail corpus (ham, hard ham, spam)Licence: No licence given by the source
Real mail received 2002 to 2005, sorted by hand into spam and non-spam. No licence is given; the corpus says copyright in the messages stays with their senders.
- Nazario phishing corpusLicence: CC-BY-4.0, as the corpus states
Phishing mail collected and sorted by hand by Jose Nazario.
Changelog
- 0.1.0 · 2026-09-30Trained on Jeff v1.2 (LoRA rank 16, one epoch). Results on the Results page. Published on Hugging Face as v1.2, with its test and calibration sets. Three of the email sources give no licence.
Comments
Comments open when JeffHub launches. They will live in the registry repository's GitHub Discussions, one thread per adapter; you sign in with GitHub, and JeffHub stores no accounts.
