We chose retrieval over fine-tuning. Here's the measurement.

The obvious way to build an AI classifier for customs is to fine-tune a model on customs rulings.

That's what the ATLAS benchmark did. LLaMA-3.3-70B, roughly 18,000 US CBP rulings, chain-of-thought supervision. And it works: 40.0 percent fully correct at ten digits, 57.5 percent at six on their held-out split. That's a large lift over frontier models answering zero-shot, which the same paper measured at 25.0 percent at ten digits.

We took a different approach and measured it on their split.

What we ran

A training-free, open-book pipeline. BM25 retrieval over the same public ruling corpus at inference time, then a rerank step, then a commodity model instructed to apply the GRI to the grounded candidates it was handed.

No training. No weights of ours. Nothing in the system that couldn't be rebuilt from public data in an afternoon by someone who knew what they were doing.

On the published ATLAS test split: 55.0 percent exact at ten digits, 66.5 percent at six.

For scale at the other end, the untuned base model — same family, no retrieval, no scaffolding — scores 2.1 percent at ten digits and 20.7 percent at six. Almost everything in that gap is the system around the model, not the model.

Now the caveats, because they're the point

This is not a like-for-like leaderboard result and we're not going to pretend it is.

ATLAS is closed-book. The model answers from what it absorbed during training, with no access to the corpus at inference. Ours is open-book: it retrieves from that corpus while answering. Those are different games and the closed-book one is harder. Anybody presenting 55 against 40 as a win has either not read the setup or is hoping you haven't.

Our argument is that the open-book setting is the operationally relevant one, because in production the rulings genuinely are available. Nobody classifying goods for money is denied access to the tariff schedule. But that's an argument about which measurement matters, not a claim to have won a fair race.

The benchmark also leaks. We found 31 of the 200 test descriptions appearing verbatim in the training split with identical labels — 15.5 percent. On the remaining 169 rows we score 48.5 percent, still above the published full-set figure, but the honest comparison would be de-leaked against de-leaked and we can't compute that. The artifacts needed to reproduce ATLAS's score on a filtered set aren't public. That's its own post, in a couple of weeks.

Why retrieval wins here

Not because retrieval beats fine-tuning in general. It doesn't, and anyone telling you there's a universal answer is selling one of the two.

It wins on this task because the correct answer to a classification question is a citation.

Tariff classification is a legal determination governed by published headings, legally binding notes and prior decisions. The useful output isn't a code, it's a code plus the authority that puts it there. A fine-tuned model has dissolved that corpus into its weights and cannot hand you the ruling it reasoned from. What it can do is produce a plausible-sounding justification that references a ruling number, which is a strictly worse outcome than saying nothing, because now somebody has to go and check whether that ruling exists and says what the model claims.

The second reason is unglamorous. The schedule changes. Annual revisions, mid-year corrections, whatever trade policy does that quarter, plus a full WCO revision every five years. A fine-tuned model is a snapshot that begins drifting the day training finishes, and there's no cheap way to tell how far it has drifted. A retrieval pipeline over a corpus you keep current is current, and refreshing it is an ingest job rather than a training run.

The third is jurisdictional. A model trained on US CROSS rulings knows US practice, and US practice only. Our pipeline runs unchanged against another schedule provided you can give it a ruling corpus for that jurisdiction — EBTI for the EU, ATaR for the UK. That turns "support a new country" from a modelling project into a data-acquisition problem, and data-acquisition problems have known costs.

There's a fourth that only shows up once you're operating it. When retrieval is wrong you can see why. The candidates that came back are inspectable, the rerank ordering is inspectable, and the failure is usually attributable to a specific bad retrieval you can then fix with a query change or a corpus addition. When a fine-tuned model is wrong, the debugging story is "collect more examples and train again," which is a much longer loop.

Where fine-tuning is right

To be clear about the other side of the line, because we tune models too.

Reading a scanned commercial invoice into structured fields. Deciding what an ops email is asking for. Normalising a carrier's event codes into your own vocabulary. Extracting components from a free-text address. None of those have a published authority. There is no ruling that governs where the consignee sits on a forwarder's invoice template. There's nothing to cite, and the task is pattern recognition over a distribution you own — which is exactly what tuning is good at.

So the rule we work by now: retrieve where a citable authority exists, tune where none does.

Classification, origin determination, sanctions screening, licensing requirements — published corpora, answer is a citation, retrieve. Document extraction, intent routing, event normalisation, entity resolution — no external authority, answer is a judgement about your own data, tune.

Most of the interesting engineering in this domain turns out to be deciding which side of that line a given task falls on. Get it wrong in one direction and you spend six months building a training set for something a lookup would have solved. Get it wrong in the other and you ship a system that confidently cites rulings it invented.

Full methodology, all the numbers and every caveat are in the HTS classification report.