BL-D-006: The Data Integrity Protocol
BIJECTIVE LABS
BL-D-006: The Data Integrity Protocol
Status: RELEASED
Classification: PUBLIC
Revision: 1
Date: 2026-09-07
1. The governing principle
A wrong number is more dangerous than a missing number.
A missing number stops the pipeline. Somebody notices, somebody investigates, and the cost is an afternoon. A wrong number produces a confident verdict, and the verdict propagates into decisions that get made weeks later by people who have no reason to doubt it.
This protocol exists because the programme repeatedly made correct decisions on incorrect inputs, and could not tell the difference from the inside.
2. The failure classes
Eleven catalogued incidents in the first programme resolve into six root causes. The classes are more useful than the individual incidents, because they predict where the next one will come from.
| Class | Failure |
|---|---|
| K1 | A missing value defaulted to zero, where zero is meaningful |
| K2 | Verified once, assumed verified forever |
| K3 | A safety filter depending on a single upstream field that can be absent |
| K4 | Sample concentration in a symbol, a period, or a regime, unchecked |
| K5 | Point-in-time violation; information used before it existed |
| K6 | Unbounded resource or accounting growth |
K1 is the most damaging because it is silent. A websocket parser defaulted a missing funding rate to zero. The venue sends that field only when it changes, so nearly every incremental message overwrote a correct value with zero, in both the cache and the in-memory state. Three separate consequences followed: a funding-crowding feature was silently disabled, a strategy's accrual calculation overstated its return, and a written claim that the data had been verified live became retroactively false.
K2 is what turns a fixed bug into a recurring one. A field verified during integration is not verified in production six weeks later, after the venue has deprecated a topic or changed a schema.
K4 produced a retracted verdict. A dislocation strategy was rejected on a sample of 39,250 events, of which 87 percent came from a single low-quality listing that the universe selector had chosen precisely because it moved a lot. The verdict was withdrawn and the honest answer was that the clean sample was too small to conclude anything.
K5 produced the most expensive single error in the archive. A hedge ratio fitted on the full sample and then replayed through that same sample turned -55 basis points into +180.
3. The binding rules
Eight rules, numbered here as they appear in BL-D-009.
| ID | Rule | Class |
|---|---|---|
| R-17 | Defaulting a missing value to zero is forbidden where zero is meaningful. Carry the last known value or fail. | K1 |
| R-18 | Every input is cross-checked against a second source before a verdict is issued. | K2 |
| R-19 | No verdict without decomposition by symbol and subperiod, and evidence that every cost term was populated from a live source. | K4 |
| R-20 | Safety filters do not depend on a single field that can be absent. A denylist backs every filter. | K3 |
| R-21 | Stale or impossible data fails closed. The consumer skips rather than proceeding. | K1, K2 |
| R-22 | Before any consumer of a field is built, the field is extracted live and compared against the design assumption. | K2 |
| R-23 | Incidents are logged the day they are found. | all |
| R-24 | A comment may not claim more than has been verified. A known compromise is tagged. | all |
4. R-22 is the one that pays
Most of these rules are recovery procedures. R-22 is prevention, it costs minutes, and it has a documented save.
A strategy design assumed the venue exposed seven to fourteen days of open-interest history. Before any code was written, the field was pulled live and inspected. It held approximately one hour.
The redesign cost nothing because nothing had been built. Had the check run after implementation, the cost would have been a rewrite of the data path plus however long it took to notice that a feature keyed on multi-day accumulation was reading a single hour.
The rule generalises to a habit worth more than the rule: before you write a consumer, look at the actual bytes.
5. The failure that no rule caught
On 2026-07-16 the programme discovered that a venue had silently deprecated a liquidation websocket topic. The system had been subscribed to a dead endpoint for more than nine days, receiving zero events, while an entire strategy candidate waited on data that was never arriving.
Two things made it invisible.
The subscription failure was reported by the venue and then discarded by the client, which checked for a success field and continued past anything else without logging it. A total-silence failure produced no error anywhere.
Zero liquidation events is not obviously wrong. Liquidation cascades are rare by nature, so a quiet feed looks like a quiet market. There was no baseline that would have flagged the difference.
It surfaced only because a recorder that should have been writing files had produced none after 42 hours, and somebody went looking for the files rather than for the bug.
The remediation went beyond the topic migration. Subscribe failures are now
logged with the venue's own message text, and the incident is the reason R-18
requires cross-source verification rather than trusting a single feed's
liveness. The full case study is BL-IN-005.
6. Pre-verdict checklist
Run before any verdict is issued. This is the Data and Sample half of the
five-angle audit in BL-D-004 section 4.
- Every input passed a cross-source check against a second source.
- No cost term is zero unless zero is the verified value.
- The result decomposes by symbol without one symbol carrying it.
- The result decomposes by subperiod and survives in each.
- Every quantity entering the signal is measurable at decision time.
- Sinks are bounded; nothing in the measurement path grows without limit.
7. Pre-build checklist
Run before any new consumer of a data field is written.
- The field has been pulled live and compared against the design assumption.
- Missing values carry the last known value or fail closed. Nothing defaults to zero where zero is meaningful.
- Any safety filter depending on this field has a denylist behind it.
- Anything the consumer writes has a bound and a retention policy.
8. What this protocol costs
Both checklists together take under an hour on a typical hypothesis. Set against that: eleven catalogued incidents, one retracted verdict, one strategy blocked for more than nine days by a feed that looked alive, and one backtest result wrong by 235 basis points.
The protocol is not overhead. It is the cheapest line item in the programme.
Copyright (c) 2026 Bijective Labs. All rights reserved.