Methodology · Deep dive
Walk-forward analysis, explained
Most backtests answer the question "how would this rule have done on the past I tuned it on?" — which is no question at all. Walk-forward analysis asks the only one that matters: how does the rule do on data it has never seen, again and again, as time rolls forward? This page explains the method with the real numbers from our own system, including the verdict that embarrassed us.
The method in four steps
- Split history into folds. Divide the sample into consecutive windows — we use three folds over roughly five years of daily observations (13,872 ticker-days).
- Fit on the past, judge on the future. Within each fold, parameters are chosen using only the training span, then performance is measured only on the unseen test span that follows it.
- Roll forward and repeat. Each fold's test span becomes later folds' history. No information travels backwards.
- Grade on retention and consistency. How much of the in-sample performance survives out-of-sample (the retention ratio), and does it survive in every fold — one negative fold means the average is luck-dependent.
The pitfall almost everyone hits: annualisation
Inside a fold, the test window is much shorter than the training window — in our setup, roughly 0.43× the length. Compare raw cumulative returns across windows of different lengths and the shorter window loses automatically: we watched a perfectly consistent strategy score a 0.38 retention ratio — "weak" — purely because of window arithmetic. Annualise both legs before taking the ratio. This bug existed in a published open-source implementation we ported from; it is worth checking in any walk-forward code you rely on.
A real example: the verdict that reversed
Our momentum-only signal (RSI-based) looked excellent under the standard single 70/30 train/test split: a 1.31 retention ratio — comfortably "robust" by the usual thresholds. The same signal under 3-fold walk-forward: −0.24 retained, losing money outright in one fold. Verdict: overfitted. The single split had simply landed on a friendly test window.
Our twelve-measurement composite, same harness: ~0.35 retained — a grade of WEAK on our own scale, which we publish rather than hide — but positive in every fold. Consistency under unseen data, not a flattering average, is what we require of anything that ships. Full context on the main methodology page.
Walk-forward vs. the alternatives
| Approach | What it catches | What it misses |
|---|---|---|
| In-sample backtest | Coding errors, roughly nothing else | Everything — the rule has seen the answers |
| Single train/test split | Gross overfitting | Window luck — one friendly test span flatters, as our RSI example shows |
| Walk-forward (n folds) | Window luck, regime dependence, parameter fragility | Regimes absent from the whole sample; structural breaks still to come |
Note the last cell: walk-forward is the strongest of the three and still not proof. Five years of history contains only the regimes it contains. That limit is why no Farlens output is a recommendation — the method quantifies the past's consistency, not the future's behaviour.
Checklist for reading anyone's backtest (including ours)
- Was any parameter chosen after seeing the test data? (If unstated, assume yes.)
- Is performance reported per-fold, or only as an average that can hide a losing fold?
- Are train and test legs annualised before comparison?
- Are transaction costs and missing-data handling stated?
- Is the sample size (observations, not years) disclosed?
Related reading
- The Farlens composite methodology
- Sharpe ratio calculator — the ratio being retained in the grades above