Skip to content

A Backtesting Protocol in the Era of Machine Learning: summary and key takeaways

The paper argues that finance data are too limited for machine learning to be used without discipline. It shows a nonsense ticker-letter strategy that passes a simple train/validate split, then sets out a seven-category protocol to reduce false discoveries.

Arnott, R.D.; Harvey, C.R.; Markowitz, H. · 2018 · SSRNSummary updated 20 November 2026
Published2018Working paper
Asset focusAsset-agnosticBacktesting
TypeStrategyQuant Crises & Methods
Reading time4 minthis summary

A Backtesting Protocol in the Era of Machine Learning at a glance

The question
How should researchers structure backtests, especially machine learning ones, so that fewer false strategies reach live trading?
Dataset & period
Illustrative example: a long-short strategy on NYSE stocks, built on 1963 to 1988 and validated on 1989 through 2015. The paper cites about 55 years of quality equity data, under 700 monthly observations. Otherwise a conceptual paper with no new dataset.
Method
This is a methods and opinion paper, not an empirical study. The authors use a deliberately absurd data-mined strategy to show how overfitting survives cross-validation. They then discuss common research pitfalls and group them into seven categories of checks, summarised as a question list in Exhibit 2.

Key findings of A Backtesting Protocol in the Era of Machine Learning

  • A market-neutral strategy that sorts stocks by letters in ticker symbols was found in 1963 to 1988 and looked even stronger in 1989 through 2015, despite having no economic basis.
  • The ticker-letter example shows that a single historical split can validate a false strategy, because the split is not randomised.
  • High-quality equity data cover about 55 years, under 700 monthly observations, which the authors call far too small for most machine learning and impossibly small for deep learning.
  • Chordia, Goyal and Saretto examined 2.1 million equity strategies and found 17 that pass their statistical and economic thresholds. The authors view all 17 as having little or no economic foundation.
  • Arnott, Beck and Kalesnik found eight popular factors averaged 5.8% a year before publication and only 2.4% afterward, before fees and trading costs.
  • With 20 variables, testing a few interactions can imply 190 possible pairings rather than 22 tests, so significance claims must count all of them.

Robustness, as reported in the paper

  • The authors state that out-of-sample data are never truly out of sample, since researchers know the history. Only live trading is a true test.
  • Backfilled historical data and other countries are named as better checks, but with caution because markets are correlated across countries and researchers know the past.
  • A model that is edited after failing out of sample and then passes has been overfit, not validated.
  • Results should hold under small changes in data transformations such as volatility scaling or winsorization level.

Caveats and limits

  • The paper offers no empirical test of the protocol itself and no measurement of how many false discoveries it would prevent (inferred).
  • The authors say the list is long but not exhaustive, and that balancing false discoveries against missed good strategies is left to ongoing research.
  • The protocol is qualitative. It gives no numeric thresholds for how much to adjust significance for multiple testing (inferred).
  • The ticker-letter exhibit is an illustration. Its performance figures are in an image that is not readable in the extracted text.

Items marked (inferred) are our reading of the design, not statements by the authors.

Why it matters

The paper gives practitioners a checklist for reviewing backtests, whether built with simple sorts or machine learning. It stresses recording every test tried, fixing data choices in advance, counting costs, and preferring simple, interpretable models. It is useful when evaluating a strategy pitch or an internal research process.

Questions about this paper

Why can cross-validation fail in finance?

Financial data are limited, mostly monthly or quarterly, and a single non-randomised split can still validate a false strategy, as the ticker-letter example shows.

Why is a t-statistic of 2.0 not enough?

Given 20 randomly selected strategies, one will likely clear that bar by chance, so the threshold must reflect the number of tests tried.

What does the paper say about interactions between variables?

Starting with 20 variables, interactions imply 190 possible pairings, and any significance claim should account for all of them.

Where can I read the original paper?

The original is at its publisher: https://doi.org/10.2139/ssrn.3275654. We link to the paper and do not host the PDF.

Follow new quant research

Download MacroFinance market data as CSV or Excel, or pull it by API, and test the ideas in these papers yourself. New summaries are added as we review them.

Create a free account
Summaries are written by MacroFinance from the published papers, in our own words, for information only. They are not investment advice and may omit detail. Papers are not hosted here: each page links to the original at its publisher.