Model Reasoning Benchmarks: Our Own, Self-Contained
A benchmark is only useful if the model has not already seen the answers. That sentence sounds obvious. It is also, in practice, violated by every major public benchmark in the LLM ecosystem. MMLU, HellaSwag, GSM8K, TruthfulQA, HumanEval — these datasets have been circulating online for years, ingested into the pretraining corpora that produced the models now being evaluated on them.[3] The result is a measurement system where the student has already taken the test. S10.1 established prediction markets as sensors. S10.2 built the research pipeline from arxiv to synthesis. S10.3 added blogwatcher as the early-warning layer. S10.4 grounded citations in mechanical attribution. This article addresses a gap that none of those tools solve: how do you know whether the model you are relying on actually reasons, or whether it is just reciting?
The contamination problem
Benchmark data contamination is not a theoretical concern. A 2025 survey from researchers analyzing contamination across major LLM benchmarks found that it is “a widespread and prevalent issue that undermines reliable evaluation of LLMs.”[3] When benchmark samples appear in pretraining corpora — directly or through indirect channels like distillation and teacher forcing — the resulting accuracy scores reflect memorization, not generalization.[1] A model that scores 92% on MMLU may have encountered those exact questions during training. The score is real. The capability it claims to measure is not.
The contamination is hard to detect at scale. As corpus sizes grow into trillions of tokens, decontamination methods become increasingly inadequate at identifying leaked instances.[1] Researchers have proposed dynamic benchmarking — constantly updating the test set to stay ahead of contamination — but this introduces a different problem: a shifting target makes it impossible to maintain a consistent baseline for tracking model progress over time.[1][2] You cannot compare this year’s model against last year’s model if last year’s test no longer exists.
The industry response has been to build harder benchmarks. ARC, GPQA, HLE — each new benchmark raises the ceiling so that even contaminated models cannot score near-perfect.[5] This works until it does not. The pattern repeats: a new benchmark appears, models improve, the benchmark becomes saturated, contamination catches up, and the community builds another one. It is an arms race with no stable equilibrium.
The case for private test sets
The alternative is straightforward: build your own benchmark, keep it private, and never publish the test data.[4] This eliminates contamination by construction. If the test set has never been online, no pretraining corpus contains it. If no pretraining corpus contains it, the model must actually reason to score well.
Private test sets are not new. Organizations in regulated industries — medical, legal, financial — have been building custom evaluation datasets for years.[4] The practice is becoming more common as LLMs move into production. The recommendation from practitioners is specific: supplement public benchmarks with 100 to 200 test cases that represent your actual workload.[6] Generic benchmarks tell you how a model performs on academic tasks. Custom benchmarks tell you how a model performs on your tasks.
The design of a private benchmark matters more than its size. A 200-question multiple-choice test that mirrors your domain, your data formats, and your failure modes is more informative than a 10,000-question public benchmark that tests general knowledge.[4] The questions should reflect the reasoning patterns your system actually requires: multi-step deduction over retrieved documents, factual grounding across conflicting sources, structured output from ambiguous input, graceful degradation when context is insufficient.
Building a self-contained evaluation
The Kingdom of Truth’s approach to model evaluation is self-contained by design. The wiki’s time-capsule methodology already implements a form of custom benchmarking: structured question suites scored on specificity, correctness, and novelty, submitted to models under controlled conditions.[7] The scoring protocol weights correctness at 50%, novelty at 30%, and specificity at 20% — deliberately prioritizing novel reasoning over rote accuracy.[7] This is not a public leaderboard game. It is an internal measurement system designed to answer the question that matters: which model produces the most useful reasoning for our specific use case?
The self-contained approach extends beyond the time-capsule framework. When the Council evaluates models for specific roles — OpenClaw for creative work, OpenFang for security analysis, ZeroClaw for infrastructure — the evaluation is grounded in the actual tasks each model will perform. A creative model is tested on prompt adherence, stylistic consistency, and brand coherence. A security model is tested on threat identification, vulnerability analysis, and false-positive rates. An infrastructure model is tested on code generation, debugging accuracy, and architectural reasoning.
These are not MMLU categories. They are domain-specific evaluation criteria derived from the production workload. The test cases come from real scenarios the system has encountered — actual creative briefs, actual security incidents, actual infrastructure challenges. The test set grows as the system grows. Every significant task produces new evaluation material. The benchmark is not a static artifact — it is a living reflection of the system’s actual requirements.
What this means for model selection
The practical implication is that model selection should not be driven by public leaderboard scores. Those scores are contaminated, they measure the wrong things, and they do not reflect your workload.[4][5] A model that scores highest on MMLU may be the worst choice for your specific domain. A model that scores lower on public benchmarks may outperform on your custom test set because it reasons better in your context.
For the S10 series, this connects to the broader market sensing stack. Prediction markets tell you what the crowd believes. The research pipeline tells you what the evidence says. Blogwatcher tells you what independent analysts think. Grounded citations tell you what you can prove. None of these tools tell you whether the model processing all of this information is actually reasoning about it or just pattern-matching against training data.[3] A custom benchmark is the missing validation layer. It is the sensor that measures the sensor — the test that confirms your model is doing what you think it is doing.
The Kingdom of Truth’s architecture treats model evaluation as infrastructure, not as a one-time assessment. The benchmark lives in the vault. The test cases are versioned. The results are tracked over time. When a new model is released, it runs against the existing test suite before it is considered for any role. When the system’s requirements change, the test suite updates. The evaluation is self-contained, continuously maintained, and grounded in the actual work the system performs.
This is not a rejection of public benchmarks. They have their place in comparative research and community evaluation. But for a sovereign system that depends on model reliability for critical operations — creative generation, security analysis, infrastructure management — the evaluation must be internal, private, and grounded in real tasks. The benchmark is not a leaderboard entry. It is a trust mechanism.
Sources
- LLM Benchmark Datasets Should Be Contamination-Resistant
- How Can I Publish My LLM Benchmark Without Giving the True Answers Away
- Benchmark Data Contamination of Large Language Models: A Survey
- Disadvantages of Standard LLM Benchmarks
- 30 LLM Evaluation Benchmarks and How They Work
- Benchmarking LLMs for Business Applications 2025
- Council-Vault wiki concept: time-capsule-methodology
Seventh article in the S10 series on Market Sensing & Research. Design notes on a running system, not a sales pitch.
