BL-D-010: Minimising the Gap Between Backtest, Shadow, and Live

Status: RELEASED

Classification: PUBLIC

Revision: 1

Date: 2026-09-07

---

1. The gap is not one thing

"Backtest results do not survive contact with the market" is true, universally believed, and useless as guidance, because it names no mechanism and therefore suggests no fix.

The gap is a stack of specific, individually measurable assumption failures. Each one has a name, a magnitude, and a remedy. This article enumerates the ones this programme actually encountered, with what each cost.

The three environments differ in what they can test:

The middle row is the important one. Shadow closes most of the gap for the price of calendar time and no capital, which is why the falsification chain puts it at layer 7 and why nothing in this programme was permitted to skip it.

---

2. Seven sources of gap, measured

2.1 The fill that never happens

A passive strategy was backtested against historical spread data showing regular widening events. Deployed to a live venue for 18 hours, it produced 823 triggers and zero fills.

The backtest had asked whether the spread widened. The market answered a different question: whether a quote posted at that moment, behind an existing queue, would be reached before the spread closed. It was not, ever.

Remedy: a fill assumption is a claim requiring evidence, not a modelling convenience. Where a strategy depends on being filled passively, fill probability is measured as its own quantity before the strategy is evaluated.

2.2 The fill that happens too early

An early tick-exact backtest filled entries at the price that triggered the signal. Real execution arrives after network latency, and the price at arrival is not the price that fired.

Remedy: entries fill at the real tick following the measured latency, not at the trigger tick. On this envelope that is 5 to 6 milliseconds on a persistent connection, and the correction was applied to every subsequent evaluation.

2.3 The fill that was missed by the matcher

A passive-fill replay used a first-in-first-out queue model to decide which resting orders would have been filled. An adversarial audit found the matcher was not counting trade-through fills, where a trade sweeps past a resting level without a discrete event at that level. At one instrument, 55.7 percent of adverse fills were being missed.

Every fill the matcher missed was an adverse one, so the reported gross return was an optimistic upper bound rather than an estimate.

Remedy: the matcher was corrected with explicit trade-through handling and eleven tests. The original verdict was re-run and upheld, with worse numbers than before. That direction matters: an audit that only ever improves your results is not an audit.

2.4 The cost that was never in the calculation

Realised return was computed as the difference between entry and exit mid-price. No fee, no spread, no slippage, no funding. Corrected, the expectation moved from marginally positive to -5.6 basis points.

This is the largest single gap in the programme's history and it was not a modelling subtlety. It was an absent term. Documented in BL-IN-002.

Remedy: R-01, net from day one, and the cost identity in BL-D-011 section 4.

2.5 The parameter fitted on the future

A hedge ratio estimated on the full sample and then replayed through that same sample produced a backtest return of +180 basis points at an 80 percent win rate. Re-estimated strictly on trailing data, the same strategy returned -55 to -68 basis points with a t-statistic at or below -4.

Remedy: BL-D-011 section 3. Parameters are part of the signal and must be measurable at decision time. A walk-forward harness with the estimation window enforced in code replaced the ad-hoc backtest.

2.6 The substrate that differed between shadow and live

A strategy consumed market data from a streaming feed in shadow and from a REST client in its intended live path. The two disagreed in coverage, in timing, and in which fields were populated, so the shadow record was measuring a different strategy from the one that would run.

Remedy: shadow and live share one substrate and one code path. The strategy was rebuilt to use the same client in both. This is the least glamorous item on the list and one of the most important, because the discrepancy is invisible from inside either environment.

2.7 The universe that was not the universe

A verdict was computed on a sample where 87 percent of events came from a single low-quality listing, selected by a filter that ranked on movement and therefore preferred exactly the instruments a live strategy would refuse to trade.

Remedy: the tradeable universe is declared, versioned, and identical between evaluation and execution, with a denylist behind the automated filter.

---

3. The only strategy measured in all three environments

One strategy in this record has a backtest figure, an out-of-sample figure, and a live figure. It is the cleanest evidence available on whether these practices work.

The backtest missed the live outcome by more than 200 basis points. The walk-forward figure missed it by roughly 30 to 40, in the conservative direction.

That is the argument in one table. The remedy did not make the strategy work, because the strategy did not work. It made the measurement honest, and an honest negative measurement is what stops capital from being committed.

---

4. The gap that remains

Three things shadow cannot tell you, and this record does not claim otherwise.

Whether your order fills. Shadow places no orders. Fill probability is estimated, and every estimate in this record is labelled as one.

Market impact. A strategy running at size moves the book it is measuring against. Nothing in this programme operated at a size where that was testable.

Your own behaviour under loss. Not a measurement problem, and out of scope for a research record, but it is the remaining gap between a validated shadow result and a live deployment, and it should be named rather than ignored.

---

5. Practice

The last line is the one that costs calendar time and cannot be shortened. It is also the only line that has ever caught something the others missed.

---

Copyright (c) 2026 Bijective Labs. All rights reserved.