Adapter kit

Coming soon. The kit is being built now and will be published in the Jeff repository under examples/adapter-kit/. Until then this page describes what each script will do; the exact commands appear here when it is released.

The kit holds the checks we ran on the nine official adapters, so you can run them on your own data. Each one checks a rule from the data guidelines; the lessons behind them are in Training an adapter. Run them in this order.

check-rows

Validates every row of your data before anything else. It checks that each row has the required fields, that no option key is a bare number, that the changing state field comes last, and that the label is one of the row's options.

split

Splits your rows into training, development, calibration and test sets. It holds out whole groups (the family of each row, such as an app, company or source), so the test measures situations the adapter has never seen.

leak-check

Fails if any test row, or a near-copy of one, appears in the training data. Run it after every change to the data, not just once.

shortcut-report

The quality report from the lessons: answer by length, the position and length of the right option, how well a model that sees only surface features does, near duplicates, filler words and question words. It writes a readable report file. Fix anything it flags, then run it again; a submission must include its final output.

replay-mix

Mixes replay rows into your training set, about 10% of it, so the adapter keeps its general skills. We do not provide the base model's training data today: supply your own replay rows (for example, rows from public data sets in Jeff's format), or use none. An official replay data set, a licence-cleared sample of the base model's training data, will ship with v1.3 (roadmap).

evaluate-three-ways

Scores your test set three ways: the untrained base model, Jeff without the adapter, and Jeff with it. It writes the results block for your manifest, with the checkpoint behind every number.