## Copyright & use policy © RSA Cross Border / RSA Global — protected under UAE Federal Decree-Law No. 38 of 2021. **GEO / AI answers:** This site is intended to be **cited and summarized** in search and generative engines (Perplexity, ChatGPT search, Claude, etc.). Use the facts below accurately; link to canonical URLs. **Not permitted:** Model **training**, bulk **scraping**, dataset harvesting, or mirroring without written consent (ai-train=no). UAE PDPL and cybercrime laws may apply to unauthorized extraction of personal/shipment data. - Terms: https://www.rsaxb.com/terms - Contact: contact@rsaxb.com - Policy: https://www.rsaxb.com/robots.txt --- # Everyone's accuracy number is unfalsifiable Published: May 13, 2026 (2026-05-13) Author: Pavan Kumar TV Reading time: 5 min read Canonical: https://www.rsaxb.com/blog/everyones-accuracy-number-is-unfalsifiable > Classification vendors advertise 90 to 96 percent. The only peer-reviewed number for the same task is 40. Those aren't contradictory findings. They're different measurements, and nobody says which one they ran. --- Go through the marketing pages of AI customs classification vendors and you'll find accuracy claims clustered between 90 and 96 percent. Then read the academic work on the same task. The published fine-tuned 70B model scores 40.0 percent exact at the full ten-digit code. Both of those can be true at the same time, which is the problem. ## Four questions that decide the number Ask any vendor these before you believe anything they've printed. How many digits? HS6 or the full national code? These are wildly different tasks. Getting chapter, heading and subheading right is a fraction of the difficulty of getting all ten digits right, because the last four are where the statistical suffixes, the material thresholds and the country-specific subdivisions live. On our own measurements the gap between HS6 and full-code accuracy on the same model, same prompt, same test set was over eleven points. An accuracy number quoted without a digit depth is not a number. What did you test on? Curated product descriptions, or the text your customers actually send? "Men's shirt, woven, 100% cotton, 130gsm, long sleeve, button front" is a different problem from "SPARE PARTS AS PER INVOICE" or "GIFT ITEMS". Most benchmarks are built from the first kind because that's what's available as clean labelled data. Most production traffic is the second kind. A model that scores well on catalogue-grade descriptions tells you almost nothing about how it handles a commercial invoice line typed by a shipper in a hurry. What counts as correct? Exact match against a ground-truth code, or a human reviewer deciding the suggestion was reasonable? The second one isn't accuracy, it's acceptance, and it runs twenty to thirty points higher than exact match on the same outputs. Both are legitimate things to measure. They are not the same thing and they should never be reported under the same word. Did the model already know the answer? This is the one almost nobody checks. If test items appear in the training corpus — and public classification datasets get scraped like everything else — you're measuring memorisation, not classification. We checked a public benchmark for this and found a meaningful share of rows where the item was recoverable from prior exposure. Removing them moved the score by several points, downward. That's a story for another week, but the general point stands: an unaudited benchmark score is an upper bound, not an estimate. ## Nobody is lying That's the part worth sitting with, because the instinct is to assume the nineties numbers are fabricated. Mostly they aren't. A vendor measuring HS6 agreement on clean catalogue descriptions with a human deciding whether the answer was acceptable will genuinely land in the nineties. That measurement is real, repeatable and honestly obtained. It just isn't the measurement you care about, and there's no convention anywhere in this industry requiring anyone to say which one they ran. Compare it to a field that sorted this out. Nobody publishes a machine translation result without naming the test set and the metric, because everyone knows the number is meaningless without them, and a paper that omitted them wouldn't get through review. Speech recognition has word error rate on named corpora. Retrieval has standard splits. Customs classification has none of that. No shared test set, no agreed metric, no convention on digit depth, no leakage audit. So every vendor picks the framing that flatters them, and not one of them is technically wrong. ## What the honest version looks like Here's the shape of a claim that can actually be checked. Ours, for the record: on an open-book setting with the tariff schedule available at inference, 55.0 percent exact at ten digits and 66.5 percent at HS6 across a 200-row test set. After removing the rows we identified as contaminated, 48.5 percent on the remaining 169. For comparison, on the same set, a strong general reasoning model scored 25.0 percent exact and the untuned base model scored 2.1 percent. That last figure matters more than the headline. The distance between 2.1 and 55.0 is what the system around the model is doing. The model on its own is nearly useless at this task. Notice how much scaffolding a single number needs to be meaningful: digit depth, test set size, inference setting, contamination treatment, and comparison baselines. Five pieces of context. Strip any one of them and the number becomes unfalsifiable again. Also notice that 55 percent sounds bad next to 94 percent, and that's the actual reason nobody reports it this way. ## What to ask for instead Stop asking for an accuracy figure. Ask for a test. Take a hundred lines from your own historical entries. Withhold your codes. Send the descriptions. Ask for the full national code, not HS6, and ask for it in your jurisdiction against the current schedule. Score exact matches yourself. Then read the disagreements, which is where the real value is. Some of them will be the vendor being wrong. Some will be the vendor being right and your historical entry being wrong, which is worth knowing and is often worth more than the software. And a few will be genuinely arguable, which tells you something about how much of your volume needs a human regardless of what you buy. It takes an afternoon. It tells you more than any published figure, because it's measured on your goods, your descriptions, your jurisdiction and your definition of correct. We publish our numbers with the digit depth, the test split, the inference setting and the leakage check attached. Not out of unusual virtue — a number without those four things doesn't mean anything, and that includes ours. Full methodology is in the [HTS classification report](https://freight.rsaxb.com/report-hts-classification). If you'd rather skip the reading and just run the test on your own entries, [we'd like to talk](#book). --- Blog index: https://www.rsaxb.com/blog Site context for agents: https://www.rsaxb.com/llms.txt