The S8 series has built a case for simulation. S8.1 introduced the
sim loop — a paper-trading engine that replays resolved markets against
real trade tapes, with slippage, fees, and KOT scoring. S8.3 defined the
promotion gate — the criteria a strategy must clear before it touches
real money. S8.4 formalized KOT-scored edges. S8.6 built the dashboard
feedback loop. Each article assumed the simulation works, then showed
what to do with the result.
This final article asks the harder question: when does the simulation
lie?
Not because it is broken. Not because the code has a bug. But because
every simulation is a model of reality, and every model has a boundary
beyond which its predictions degrade. Knowing where that boundary sits
is the difference between a useful tool and a dangerous one.
The structural
limit: models model the past
limit: models model the past
The simulation-mode engine replays resolved Polymarket markets using
their real trade tapes. Every fill executes at the tape price at the
exact timestamp a signal fires. The engine sees zero lookahead — only
prints up to the current tick. This is its strength: it tests strategies
against what actually happened, not what someone imagines might
happen.
But “what actually happened” is a sample. A finite, biased,
historically contingent sample. The dual_trend strategy earned
PROVEN_EDGE with 40 trades and a profit factor of 5.9. That is a real
result. It is also a result drawn from a specific corpus of resolved
markets — markets that existed in a specific economic regime, with
specific liquidity conditions, specific participant behavior, and
specific volatility patterns.
The simulation does not know this. It reports the numbers honestly.
But the numbers describe performance on that corpus, under
those conditions. Extrapolating from 40 historical trades to
“this strategy will make money in the future” requires an assumption
that the future will resemble the past. Sometimes it does. Sometimes it
does not.
Overfitting:
when the sim sees patterns that are not there
when the sim sees patterns that are not there
Overfitting is the most common failure mode in any simulation-based
system. It happens when a strategy learns the noise in the historical
data rather than the signal. The strategy fits the past perfectly — and
fails on new data because the patterns it learned were artifacts of the
sample, not features of the market.
The simulation engine mitigates this through KOT scoring’s minimum
sample size requirement. A strategy needs at least N trades before it
earns a verdict. The dual-side momentum strategy needed 40 trades. This
is a guard against overfitting on thin data — a strategy that wins 3 out
of 3 trades might be lucky, not skilled.
But minimum sample size is a necessary condition, not a sufficient
one. A strategy can pass the sample size gate and still be overfitted if
the corpus itself is narrow. Forty trades across a specific set of
Polymarket markets is not the same as forty trades across diverse market
conditions, time periods, and asset classes. The simulation reports what
it tested. It does not report what it did not test.
The honest position: the sim loop proves a strategy survived
historical replay. It does not prove the strategy will survive the
future. The promotion gate in S8.3 added paper-trading and shadow mode
as staging steps precisely because historical survival is the
first filter, not the last.
Regime change:
when the rules shift under the sim
when the rules shift under the sim
A regime change is a structural shift in the environment that
invalidates the assumptions a model was built on. In prediction markets,
regime changes include: a major regulatory action that alters
participant behavior, a liquidity crisis that widens spreads, a new
class of market participants (institutional vs retail), or a fundamental
shift in the underlying asset’s volatility profile.
The simulation engine cannot detect regime changes. It replays
historical data. If a regime change occurs after the historical period
the sim tested against, the strategy’s edge may evaporate — or reverse.
The edge formula (edge = p_true - market_price) depends on
the relationship between spot momentum and prediction market pricing. If
that relationship changes — if, for example, a new class of participants
prices crypto prediction markets more efficiently — the edge shrinks or
disappears. The sim would still show the old edge as PROVEN_EDGE because
it tested against the old relationship.
This is not a flaw in the simulation. It is a property of all models.
The dashboard feedback loop from S8.6 addresses this by providing
continuous monitoring: when live performance diverges from simulated
performance, the promotion gate can reverse. But the monitoring only
catches divergence after it appears in live data. It does not
predict regime changes in advance.
The honest position: simulations are backward-looking by design. They
test whether a strategy would have worked. They cannot
guarantee it will work. The gap between “would have” and “will”
is filled by monitoring, not by more simulation.
Black swans and tail risk
Nassim Taleb’s observation applies directly: the most consequential
events are the ones the model never saw. A strategy tested against 40
Polymarket markets has never been tested against a market that goes to
zero in a single block, a flash crash that wipes out all liquidity, or a
coordinated manipulation event. These are not theoretical risks — they
are features of prediction markets, which are thinner and more volatile
than traditional financial markets.
The simulation engine’s slippage model (25 basis points default) is a
reasonable approximation for normal market conditions. It is not a model
for tail events. A flash crash can produce slippage orders of magnitude
larger than 25 basis points. The simulation would report a clean fill at
the tape price; the real execution would have experienced severe adverse
movement.
The honest position: the simulation models the expected case. It does
not model the catastrophic case. Risk management — position sizing via
Kelly Criterion at quarter-Kelly, per-market entry caps, flat-stake
sizing — is the layer that handles tail risk. The simulation tests
whether the strategy works under normal conditions. The sizing rules
ensure that when conditions are abnormal, the damage is contained.
Where the simulation
actually works
actually works
Despite these limits, the simulation loop is not merely a warm-up
exercise. It has proven value in specific, well-defined areas:
Strategy elimination. The sim killed the naive
YES-only strategies cheaply. These strategies lost money on historical
replay — not because of regime change or black swans, but because the
underlying logic was flawed. The sim catches strategies that are wrong
in principle, not just in practice.
Execution quality testing. The interleaved-fill bug
from the initial run was a real execution problem that unit testing
could not catch. The sim exposed it because it replays the actual
texture of market data. This is a genuine advantage over theoretical
analysis.
Baseline establishment. A strategy that passes KOT
scoring provides a baseline expectation: “under these historical
conditions, this strategy produced this result.” That baseline is
valuable even if the future differs from the past — it gives you a
starting point for monitoring divergence.
Decision recording. The task tracker records which
strategies were tested, what the verdict was, and why. This
institutional memory prevents re-testing the same failed strategy under
a different name — a common failure mode in discretionary trading.
The honest boundary
The simulation is a filter, not a crystal ball. It removes strategies
that are clearly flawed, establishes baselines for strategies that
survive, and creates a recorded decision trail. It does this well. What
it cannot do is guarantee future performance, predict regime changes, or
model tail events.
The S8 series built a pipeline: signal detection
(edge-detection-cli), simulation (simulation-mode engine), scoring (KOT
scoring), promotion gate (S8.3), paper trading (S8.5), dashboard
monitoring (S8.6). Each layer adds a checkpoint. The simulation is the
first and most important checkpoint — but it is only the first.
The dashboard and task tracker complete the picture. The dashboard
shows what is happening now. The task tracker records what was decided
and why. Together with the simulation, they form a feedback loop:
simulate, score, gate, monitor, record, repeat. The loop does not
eliminate risk. It makes risk visible, measurable, and reversible.
Trust the simulation when it tells you a strategy is flawed — it is
usually right. Trust the simulation when it tells you a strategy will
make money — it might be right, but verify with paper trading, shadow
mode, and live monitoring before committing capital. The simulation
earns trust through the quality of its elimination, not the certainty of
its predictions.
That is the honest limit. And knowing it is what makes the simulation
useful.
