Why I'd Reject a Strategy That Looks Great in a Backtest (5 Robustness Tests)

You build a beautiful strategy, fine-tune it on past data, launch it live, and it loses money. A lot of it. The cause is almost always curve fitting, and the only defense I know is strict robustness testing. The BreakoutOS Robustness Suite runs five proven tests and balances them into one score from 0 to 100. In this post I'll walk through all five on a real strategy that scored 74 overall, and show you why I still wouldn't trade it.

Why Great Backtests Fail in Live Trading

For a lot of traders, the journey looks like this. They build a strategy, tune it until the backtest looks perfect, and switch it on. Then it collapses. This happens to beginners, and it happens to traders who've been at it for years.

The reason is a well-known problem called curve fitting, or overfitting. It's very easy to take past data and create conditions that fit that data perfectly. It's far harder to know which strategy will keep working the same way on unseen, out-of-sample data.

This problem is as old as trading itself, and the vast majority of strategies are overfit. The only solution is a strict set of robustness testing procedures that separates the overfits from the strategies with a potentially genuine edge.

There are plenty of techniques to choose from, starting with a simple in-sample and out-of-sample split and going up to full walk-forward analysis. The trouble is that many popular retail trading platforms ship with very little robustness tooling, or none at all. So most traders go live with strategies that were never properly challenged.

Five Tests, One Score: What the Robustness Suite Does

The Robustness Suite is part of BreakoutOS, a platform built for one job: developing breakout strategies on any market or timeframe. It was built by breakout trading professionals who took years of hedge fund experience and their best algorithms and put them into one place.

The suite scores strategies you build inside the platform. You generate a list of prototype breakout strategies, click on any one of them, and the full robustness breakdown is there with the rest of the strategy details.

What makes it different is that it combines five robustness tests into one. Each test is already a proven, standalone procedure. The suite runs all five, balances them, and delivers a final score that tells you how prepared the strategy is for live trading.

And the score works. We ran a study on 103 strategies across 8 markets, scoring each one on in-sample data only. Of the strategies that scored 65 or higher, 68% stayed positive on data the score had never seen, against 48% below that line. Above 80, it was 76.2%.

The full study, including the market-by-market breakdown, is in The Score That Predicts Which Strategies Will Work Live.

The 5 Robustness Tests, One by One

Here's what each test measures, using the scores from the example strategy in the video.

1. Space Robustness (score: 80)

This is the most essential test of the five. Every breakout strategy is defined by a point of initiation and a multiplier of volatility, which I call the space. Space Robustness asks a simple question: across all the different space multipliers, what percentage would produce a positive result?

You can see it on the chart. Each gray equity curve is a different multiplier, and the suite counts how many end up positive. In this example it was 80%, so the score is 80 out of 100.

The rule of thumb is that the result simply needs to be positive. I don't care much about how much money each multiplier would've made. The more multipliers that work, the more robust the breakout model is.

This is one of the big advantages of breakout trading. Because every strategy shares this structure, you can build better tools to score it. Only a breakout approach gives you a Space Robustness Score.

2. Neighbor Sensitivity (score: 71)

When the platform recommends an ideal multiplier, the values around it have to produce positive results too. In the example, the recommended multiplier was 0.2. Move it to 0.4 and the strategy is still profitable.

That's what you want to see. A robust strategy doesn't depend on one magic number. Several parameters have to pass for this score, and the algorithm behind it is proprietary, but the output is easy to read. This strategy held up well on its neighbor values and scored 71.

3. Recency (score: 100)

Markets change a lot. They make new all-time highs, and volatility hits new highs and new lows. The ideal state is that once you deploy a strategy, it's ready to make money right away. For that, it has to be in tune with the most recent market conditions.

The Recency score checks how ready the strategy is across three windows: the last 6 months, 12 months, and 24 months. Recency can be subjective, but those three windows work well for breakout trading, because good breakout strategies are developed on at least 10 years of data. This strategy scored 100 here.

4. Walk-Forward (score: 54)

Walk-forward analysis is the gold standard in trading. You should never even think about trading a strategy without it.

The process splits your data into many pieces. You optimize the strategy on the in-sample part, the data it's allowed to see, then confirm it on unseen out-of-sample data. And you repeat that multiple times.

Turning that into a score isn't easy. BreakoutOS uses an anchored walk-forward, starting from the beginning of the data in 2011 and adding a new block each step: 2012, 2013, 2014, and so on. For each block, it checks where the chosen parameter (here, the space multiplier) would land in the out-of-sample ranking. First, second, third. Then it takes the average rank and turns it into a score.

For this strategy the result was 54, which is mediocre. I'm very strict with the walk-forward score. A mediocre result here is a deal-breaker for me, even when the overall score looks positive.

5. Edge Quality (score: 66)

The last test is BreakoutOS's own edge quality assessment. It's built on hedge fund algorithms and combines different metrics, parameters, and functions into one measure of how good the breakout edge is.

This strategy scored 66. That's right at the threshold to pass. It isn't the best score, but it's workable.

Why I'd Reject This Strategy Despite a 74

Put the five tests together and the overall score is 74. That's pretty high, and comfortably above the recommended minimum of 65. Here's the full scorecard:

I probably wouldn't trade it, for two reasons.

1. The walk-forward score is mediocre. At 54, the settings didn't hold their rank well enough on unseen data. That alone is enough for me to pass.
2. The scores don't agree. On all the data, the strategy scores 74. Scored on the in-sample and out-of-sample periods as standalone tests, it drops to 56 and 58. I'd like to see at least 65 to 70 on each. That's a large gap.

None of this makes it a bad strategy. It's something I could add filters to, improve a little, and potentially turn into a viable strategy. But I'm picky. I'd want higher in-sample and out-of-sample scores and a higher overall score before it goes anywhere near a live account.

A near-miss can move a long way with one change. In The Simple Time-Window Hack That Took a Breakout Strategy From 44 to 76, limiting the entry window lifted the same model's robustness score by 32 points.

How to Read a Robustness Scorecard

Here's the order I go through it in.

1. Start with the overall score. 65 is the recommended minimum. Below it, move on to the next prototype.
2. Open the components. Look at all five tests. A strong average can hide one weak result.
3. Treat walk-forward as a veto. If it's mediocre, the strategy doesn't trade, whatever the overall number says.
4. Check that the scores agree. Compare the all-data score with the standalone in-sample and out-of-sample scores. I want at least 65 to 70 on each.
5. Judge space on positive results. The multipliers only need to be profitable. How much each one made matters far less than how many of them worked.
6. Improve it or drop it. A near-miss is a candidate for filters. Add them, re-score, and see if it clears the bar.

A suite like this can be the deciding factor between making money and losing money in live trading. Personally, I can't imagine developing any strategy without one.

Frequently Asked Questions