How to read a backtest before you trust it

Every trading product you will ever evaluate arrives with a chart that goes up and to the right. Ours included. The chart is the least informative part of the pitch, because a backtest is not a record of anything that happened. It is a simulation of what a set of rules would have done, run by the same people who want you to like the result. Some of those simulations are careful. Many are not. And the difference is almost never visible in the chart itself.

This is a field guide to the ways backtests flatter themselves. Most of the flattery is accidental, which makes it more common, not less. None of it takes advanced math to catch. It mostly takes knowing which questions to ask, so this piece ends with a checklist you can put to any vendor, including us. Coil (coil.trade) keeps its methodology public at /how-it-works precisely so those questions have somewhere to land.

Who was actually in the universe that day

The first question is never about the strategy. It is about the list of stocks the strategy was allowed to pick from. An index like the S&P 500 is not a fixed set of companies. Names are added and removed constantly, and the committee tends to add companies after they have already done well. If a backtest runs today's membership list against a decade of history, it is quietly testing on companies that were selected, in part, because they went on to succeed. The index did the stock picking retroactively, and the strategy takes the credit.

The honest version uses point-in-time membership: on every simulated day, the strategy can only see the names that were actually in the index on that day. The sibling problem is delisted names. Companies that went bankrupt, got acquired under duress, or faded off the exchange tend to vanish from convenient datasets, and deleting the losers makes every long strategy look smarter. We wrote a longer piece on this in survivorship bias explained, but the vendor question is short: was membership point-in-time, and do delisted names stay in the data with their real endings?

How the fills happen, and what they cost

A backtest has to decide the price at which every simulated trade executes, and the most common choice is also the least defensible: compute the signal from the day's closing prices, then fill the trade at that same close. In live trading that is impossible. You cannot act on a close before it prints. The honest default is next-open fills, where a signal computed tonight trades at tomorrow's opening price, overnight gap and all. For strategies that trade often, the difference between those two assumptions is not a rounding error. It is frequently the whole result.

Costs are the same story in slow motion. Commissions may be near zero now, but spreads, slippage, and the market impact of your own order are not. A backtest with zero modeled costs overstates every strategy, and it overstates high-turnover strategies most, because each additional trade is another chance to pay the spread. Ask what per-trade cost was assumed, and whether the assumption scales with how much the strategy trades.

Statistics that grade their own homework

Two subtler problems live in the statistics rather than the simulation. The first is overlapping windows. If a study computes twelve-month returns starting every month, adjacent observations share most of their data. The sample looks large, but the number of genuinely independent observations is much smaller, and any t-statistic built on the raw count is inflated. When a writeup calls a result highly significant, ask how many non-overlapping periods that significance actually rests on.

The second is lookahead leaking into rolling statistics. Normalize a signal by the full sample's mean and standard deviation, and every early data point has been scaled by information from the future. Pick an indicator's lookback length because it worked across the whole series, and the choice itself is a leak. Each of these bugs is small, and none of them look like cheating in the code. Together they compound into a strategy that knew the future a little bit at every step.

The tuning problem

Then there is the most human failure mode: adjusting parameters until history looks good, and reporting the result on that same history. Every strategy has knobs, and with enough turning, the knobs stop fitting the signal and start fitting the noise. Noise does not repeat. The vendor question is whether the reported results come from the data the model was tuned on, or from data it never saw, held out or walked forward in time.

The cherry-picked window is the same trick applied to the calendar. Start the test just after a crash and end it at a peak, and a mediocre rule looks inspired. Ask what happens when the start date moves a year earlier or a year later. A robust result degrades gracefully. A curve-fit one falls apart, because it was never a strategy, only a description of one particular stretch of history.

What honesty looks like, and what to ask

Here is the counterintuitive part. If a backtest shows a strategy roughly matching its benchmark for long stretches, that is usually evidence of honest work, not failure. Real edges concentrate in particular market regimes, and the rest of the time a disciplined rule looks ordinary because there is nothing to exploit. We covered why in regime concentration. A backtest that wins in every year and every environment is more likely leaking or overfit than brilliant, because markets do not hand out edges that evenly.

So here is the checklist. Put these questions to anyone selling you a strategy, and put them to us:

• Was index membership point-in-time, and do delisted names remain in the data?
• Are fills next-open or same-close, and what costs and slippage were modeled?
• Do significance claims rest on overlapping windows, and if so, how were they adjusted?
• Were any rolling statistics or parameter choices informed by the full sample?
• Were results reported on the same data the model was tuned on?
• What happens to the result when the start and end dates move?
• Where are the stretches when it did not beat the benchmark, and what did it do there?

Clear answers do not prove a strategy will work going forward. Nothing does, and trading with any tool can lose money, whatever its history says. We went deeper on how different product categories handle this evidence problem in Coil vs trading bots and signal services. What clear answers do prove is that the vendor understands how their own evidence can lie, which is the minimum bar for taking that evidence seriously. The research behind Coil's ranking is laid out at /how-it-works. Read it the same skeptical way you would read anyone's.

FAQ

Is a backtest useless, then?

No. A carefully built backtest is a legitimate way to test whether a rule had an edge historically. The problem is that careful and careless backtests produce identical-looking charts, so the value lives in the construction details, not the picture. Treat the chart as a claim and the methodology as the evidence.

What is the fastest red flag when reading a backtest?

Vagueness about fills and costs. How trades execute and what they cost are the first things an honest researcher nails down, so a vendor who cannot answer those two questions clearly usually has problems everywhere else.

Does matching the benchmark for years mean a strategy failed?

Not by itself. Most real edges are concentrated in specific market regimes, so long ordinary stretches are what honest results tend to look like. A backtest that claims to win in every environment deserves more suspicion, not less.