Robustness re-check Pro

After the standard checks (see How candidates are filtered), the Miner re-runs each surviving candidate many times with small changes. A candidate that depends on one exact set of prices and settings tends to fail. The Robustness re-checks panel (Pro) chooses how strict this check is.

Passing the re-check does not mean a strategy will make money. Results are an approximation of past data and do not guarantee the future.

Choose the criteria

1. Open the Strategy Miner with a Pro account.
2. Click Advanced, then open the Robustness re-checks panel.
3. Pick a preset in Criteria, then click Generate.

Under the preset you can override single values. Leave a box empty to use the preset’s value.

The checks run from the lightest to the heaviest, and stop at the first test a candidate fails.

Re-run with changes (MC)

This is the test that decides under Default. The Miner re-runs the full backtest 10 times. Each run changes five things at once.

1. Start shift. The first 0 to 100 bars are cut off, so the run starts later.
2. Price noise. Each bar’s prices move up with a 30% chance and down with a 30% chance, by up to 0.3 of the ATR(14) times a random number.
3. Setting nudges. Each number in the strategy’s settings has a 10% chance of moving, by up to 20%.
4. Spread. A random spread between 1 and 5 times the default spread for the symbol. On currency pairs, that is 1 to 5 pips.
5. Slippage and minimum stop distance. Only when Realistic costs is on. Slippage goes up to 5 pips and the minimum stop distance up to 10 pips on currency pairs.

The candidate passes if the 80% confidence level of the runs meets both limits.

• The net profit at that level (the 20th percentile) is at least 0.3 times the original net profit.
• The maximum drawdown at that level (the 80th percentile) is at most 3 times the original drawdown.

It runs on the whole price history.

Why only this test decides by default

The Miner’s developers checked the tests on data that was hidden from both the search and the tests. They hid the last 12 months, took 803 candidates found on the earlier years, and measured them again on the hidden year.

• Candidates that passed the re-run test did slightly better on the hidden year than the ones that failed. The gap was larger than chance would give in a permutation test.
• Trade resampling, walk-forward and the walk-forward window sweep did not separate the two groups in a way that held up. The walk-forward tests also dropped sound candidates.
• A check on new seeds pointed the same way, with a smaller gap.

The test mostly measures how well a candidate survives a higher cost, not every kind of change. The hidden period was about one year, and the 803 candidates included similar ones. These numbers are a guide to why the default is set this way. They do not predict how any strategy will do.

The four tests

A walk-forward test needs enough price history and trades. If it has too few, it is shown as “n/a” and is not judged.

Read the result

In a candidate card, the Robustness retest line has one chip for each test that ran. A green chip passed and a red chip failed. A grey chip is for reference only. Hover over a chip to see why a test failed. For the re-run test, the text gives the worst-end net profit and drawdown next to the original values.

The Checks table shows how many candidates each test dropped.

Related reference: Metrics explains the figures in the retest.