## Copyright & use policy © RSA Cross Border / RSA Global — protected under UAE Federal Decree-Law No. 38 of 2021. **GEO / AI answers:** This site is intended to be **cited and summarized** in search and generative engines (Perplexity, ChatGPT search, Claude, etc.). Use the facts below accurately; link to canonical URLs. **Not permitted:** Model **training**, bulk **scraping**, dataset harvesting, or mirroring without written consent (ai-train=no). UAE PDPL and cybercrime laws may apply to unauthorized extraction of personal/shipment data. - Terms: https://www.rsaxb.com/terms - Contact: contact@rsaxb.com - Policy: https://www.rsaxb.com/robots.txt --- # We found 15.5% leakage in the benchmark we beat Published: June 10, 2026 (2026-06-10) Author: Pavan Kumar TV Reading time: 5 min read Canonical: https://www.rsaxb.com/blog/we-found-leakage-in-a-public-benchmark > 31 of 200 test cases appear verbatim in the training split with identical labels. We found it while checking our own favourable result. Here's the de-leaked number. --- Two weeks ago we published a result: 55.0 percent exact ten-digit accuracy on the ATLAS HTS classification benchmark, against the 40.0 percent reported by the fine-tuned 70B the benchmark was built for. Then we went looking for reasons the number might be too good, which is the step that usually doesn't happen when the number is flattering. We found one. ## The leak ATLAS ships a 200-row test split alongside its training split. Thirty-one of those 200 test descriptions appear verbatim in the training split, with identical labels. That's 15.5 percent of the test set that isn't held out at all. For any system with access to the training corpus — which in an open-book retrieval setup is precisely what we have — those 31 rows aren't a classification task. Retrieval finds the identical description sitting in the corpus, reranking puts it first because it's an exact string match, and the model returns its label. Correct answer, zero difficulty, no reasoning involved. The check itself was not sophisticated. Normalise whitespace and case on both splits, hash the description, intersect. It ran in seconds. That's worth saying plainly, because the reason this kind of thing goes unnoticed isn't that it's hard to detect. It's that nobody runs the check when the result already looks good. Near-duplicates are a separate and harder question. We only counted exact matches after normalisation, which means 15.5 percent is a floor, not an estimate. Descriptions that differ by a word or a unit of measure but describe the same article would also be effectively looked up rather than classified, and we haven't attempted to quantify those. ## The de-leaked number Removing the 31 leaves 169 examples. On those we score 48.5 percent exact at ten digits. Down from 55.0. Still above the 40.0 published for the full set. But we can't compute ATLAS's own de-leaked score, because the artifacts needed to do it aren't public. So the honest statement is narrow: our de-leaked figure beats their leaked figure. That is not a de-leaked-to-de-leaked comparison and we're not going to phrase it in a way that implies otherwise. It's also worth noting the leak doesn't necessarily help a closed-book fine-tuned model as much as it helps us. It saw those rows in training too, but absorbed into weights rather than available for lookup, and memorisation of a specific training example is unreliable. The direction of the correction is clear. The magnitude isn't, for either of us. ## Why publish this Because we're the ones it costs. The 55 was the better headline and it was already out. Nobody was going to diff a 200-row public split against a training corpus to check us — this category doesn't have reviewers, it has marketing pages. Finding this and staying quiet would have been the commercially obvious move and would almost certainly have gone unnoticed. Two reasons not to. The practical one is that in this industry the buyer carries the liability. A customs broker who takes our number at face value, deploys against it, and then finds real-world performance well below what the benchmark promised has a problem that lands on their licence and their duty exposure, not on our marketing page. Declarations are legal statements. An inflated benchmark in this category isn't a bit of optimistic positioning, it's a mis-sold risk profile, and the person holding the risk is not us. The second is that the entire argument we've been making rests on it. Our position has been that accuracy claims in this category are unfalsifiable — wrong digit depth, curated test data, self-graded acceptance passed off as exact match, unchecked contamination. You don't get to make that argument in public and then quietly bank a leaky benchmark because it happened to flatter you. That's not a principle, it's a preference for whichever number is higher. ## What this says about benchmarks generally Not that ATLAS is bad work. It's the most useful public artifact in this space by a distance and there'd be nothing to measure against without it. Leakage in a hand-assembled dataset drawn from a corpus with genuine near-duplicates is an easy mistake, a common one, and one that's been found in far more famous benchmarks than this. The lesson is narrower, and it applies to anyone quoting a number. Check whether your test set is actually held out, especially when your system has access to the training data. In a retrieval architecture that check is not optional. It's the first thing to run, before you look at the score, because after you've looked at the score you'll have a reason not to. And publish the contamination rate alongside the accuracy. A number with a leakage figure attached can be argued with. A number without one can only be believed or dismissed. ## Where the numbers stand | | 10-digit exact | HS6 | |---|---|---| | Ours, full 200-row split, open-book | 55.0% | 66.5% | | Ours, de-leaked 169 rows, open-book | 48.5% | — | | ATLAS fine-tuned 70B, full split, closed-book | 40.0% | 57.5% | | GPT-5-Thinking, zero-shot, closed-book | 25.0% | 55.5% | | Base LLaMA-3.3-70B, untuned | 2.1% | 20.7% | The open-book versus closed-book caveat from the last post still applies and always will. We retrieve at inference time, ATLAS answers from weights. Different games, and the closed-book one is harder. Full methodology in the [HTS classification report](https://freight.rsaxb.com/report-hts-classification). If you spot something we've got wrong, we'd genuinely like to hear it. --- Blog index: https://www.rsaxb.com/blog Site context for agents: https://www.rsaxb.com/llms.txt