The API

Running the server

jeff-server is the HTTP API in front of one or more engine processes. It needs at least one engine already running, so start that first.

1. Start an engine process

The engine loads the base model and the named adapters once and serves decisions over a Unix socket, so several callers can share one loaded model without each waiting for its own model to load:

uv run python -m jeff.cli runtime serve --socket /path/e.sock --adapter triage --adapter spam \
    --batch-window-ms 1 --max-batch-rows 64 --queue-limit-rows 4096 --prepare-workers 4

--batch-window-ms, --max-batch-rows, --queue-limit-rows and --prepare-workers are required: how long the engine waits to collect rows into a batch, how many rows may go into one batch (every adapter's rows together), how many rows may be queued before a new request is refused as busy, and how many requests are prepared at once as they arrive (each by a thread with its own tokenizer). On an NVIDIA GPU, --max-adapters N is also required: the engine keeps room for N adapters on the GPU, allocated at start-up, and refuses to load more. So is --prefix-pool-mb MB: GPU memory for cached prompt prefixes, allocated at start-up, which must hold --max-batch-rows prefixes of 8,192 tokens at once (7,345 MB for 64 rows; 8,192 MB also holds 266 prefixes of 1,000 tokens). On a Mac or a CPU, leave both out: the command above runs as is (both flags are refused there). --adapter is repeatable; --path ID=FOLDER loads one adapter's weights from a local folder instead of downloading them.

For an engine that keeps running, add --served-file PATH, --adapters-dir DIR and --base-version ID@VERSION. The served file holds the adapters the engine serves; once it exists the engine starts from it, and --adapter is refused when that file exists. Every adapter folder under an adapters directory (a folder holding manifest.yaml and adapter_config.json) may be served, and a reload looks for new or changed folders. --base-version names the base, so an engine can start with no adapters at all. jeff runtime health --socket /path/e.sock --timeout-s 5 exits 0 once the engine is ready. Deploying jeff-server starts all of this with one script.

On an NVIDIA GPU an engine can serve an INT4 base instead of the bfloat16 one: --base FOLDER names an INT4 base folder made by jeff runtime quantize. The engine refuses it, at start-up and whenever adapters are served or reloaded, unless the folder's gate.json (written by jeff runtime gate) passed for every adapter it would serve, at that adapter's version; the error names the adapter and the command to run. Run the gate on the GPU type that will serve the base: gate.json records the GPU, but serving does not check it. Each calibrated function also needs its threshold for that INT4 base (jeff calibrate <adapter> --base FOLDER; for an adapter served from an adapters folder, jeff calibrate <adapter folder> --base FOLDER, which writes into that folder's manifest.yaml), or it is refused when it is called.

2. Point the server at it

JEFF_ENGINE_SOCKETS=/path/e.sock JEFF_MODEL_NAME=jeff-v1.3 \
    uv run jeff-server serve --host 127.0.0.1 --port 8000 --workers 1

--host, --port and --workers have no default: binding to every interface (--host 0.0.0.0) and running more than one uvicorn worker process are both deliberate choices — jeff-server serve refuses to start without all three, naming whichever is missing. Every other setting comes from the process environment, read once at start-up, not from serve's own flags, so the same environment configures every worker identically. A setting with no default below is required; the server refuses to start without it (or, for a malformed value, at all), naming the variable. A variable that is set but empty is refused too, never read as unset.

Each worker keeps its own rate limits, sign-in lockouts and request counters, so JEFF_RATE_LIMIT applies per worker: with --workers N a caller can get up to N times the rate (and N times the concurrent requests).

Settings

With no credential configured (JEFF_API_KEY, JEFF_HUB_PUBLIC_KEY and JEFF_UI_USERS all unset) the server is open: every caller is the owner. This is refused outright when JEFF_PUBLIC=1.

The adapter builder's settings (JEFF_FACTORY and the rest) are on Deploying jeff-server.

The server needs no restart when an engine restarts: calls in flight fail, and the next call reconnects (at most one attempt a second while the engine is down). New adapters require no restart either: each API worker rebuilds its list of functions when the engines report a change (or an engine restarts), so a newly reloaded or published adapter is callable at once. While a published adapter is switched on one engine at a time, calls to adapters served by every engine continue to receive answers.

What it serves

GET /openapi.json serves the committed OpenAPI description of every endpoint, request and response shape and error — the API reference page is built from it.

Next steps

• Functions and /v1/decide: the main decision endpoint.
• Errors: what a misconfigured or overloaded server answers with.