also: curve-fitting, over-optimisation, data mining bias
Tuning EA parameters too precisely to historical data, producing a strategy that backtests well but performs poorly on unseen data because it exploits noise rather than real market patterns.
Tuning the rules until they fit the past so closely that they have learned its accidents rather than its patterns. The backtest gets better while the strategy gets worse, and nothing in the report distinguishes the two.
Overfitting is the failure mode that produces beautiful evidence, which is why it survives review that a bad strategy would not. Every optimisation pass improves the reported result whether or not it improves the strategy, so a process that only ever looks at the reported result cannot tell the difference.
An optimiser reports the ten best parameter sets for a strategy. The top row is the one that gets published; the shape of the table around it is what actually says whether the edge is real.
A real edge degrades gently as parameters move away from the optimum. A fitted one falls off a cliff, and the cliff is visible before any out-of-sample data is spent.
Calculation 2.41 in-sample → 0.88 out-of-sample, while neighbours sit at breakeven
Result The 2.41 was a property of the window, not of the rules
Overfitting is diagnosed from the process that produced the result, not from the result. Ask what was tuned, on what, and what was held back.
| Range | What it means |
|---|---|
| One window, all parameters optimised, nothing reserved | The report has no evidential value about the future. It describes how well the rules can be bent to a sample. |
| Out-of-sample window reported but re-used after each adjustment | The reserve has been spent. Once results from it feed back into tuning, it is in-sample too. |
| Single untouched out-of-sample window, looked at once | The minimum standard. Interpret the gap rather than the level. |
| Walk-forward with a broad parameter plateau and cross-instrument checks | The strongest evidence available before live deployment, because the re-fitting itself was tested. |
AIStrategyMiner listings state their data window, tester model and trade count, so a reader can ask the questions above rather than take the curve on trust.
Each time a result from the reserved window informs another adjustment, that window becomes part of the training data. After a few rounds there is no out-of-sample left, and the process feels more rigorous while proving less.
Those are exactly the figures fitting inflates. An in-sample Sharpe above 3 on a retail EA is more likely to be a symptom than a reassurance.
Parameter count is one route; strategy selection is another. Choosing the best of two hundred simple strategies on the same window fits the sample just as effectively as tuning one complicated strategy.
It restores the backtest and repeats the failure, because the new fit is drawn from the same process that produced the old one. The question to answer first is whether the original edge was ever out-of-sample.
| Across the 14 EAs published here | Figure |
|---|---|
| In-sample profit factor | 1.27–1.52 |
| Listings with a profit factor below 1.00 | 0 |
| Reporting an out-of-sample block | 11 |
| Out-of-sample return | −0.57% to +3.01% |
| Out-of-sample blocks that lost money | 2 |
| Verified runs on a real-money account | 0 |
Read the last row before the first. A rolling backtest still runs on history someone could have inspected, so nothing above is out-of-sample in the sense that decides anything — and three EAs declared a reserve start date and have published no result since, which is what reporting bias looks like from the inside.
Size, not sign, is the weak point. The median in-sample record here is 362 closed trades; the median reserved block is 21. A run that short cannot separate an edge from a quiet quarter, which is why the small negatives (−0.57%, −0.47%) carry no more verdict than the positives.
The Strategy Tester’s Forward dropdown (No, 1/2, 1/3, 1/4, Custom) splits the interval for you. It cannot stop you re-reading the forward half after every optimisation pass, and re-reading it is what spends it. Set Forward to Custom with the date you stopped looking — 2026-04-01 for ten of the blocks above — and the reserve is worth only what falls inside it: 142 trades for Cairn, 40 for the next largest, single figures for two.
