BL-TR-002: The Data Substrate

Status: RELEASED

Classification: PUBLIC

Revision: 1

Date: 2026-09-07

---

1. Inventory

Measured from disk rather than from a manifest, because a manifest is a claim and a byte count is a fact [E-0038].

The Programme II research panel drawn from this is 12,970,563 rows across 154 instruments and four streams, occupying 314 MB, built with zero retries.

---

2. Zero retries is a data-integrity claim

That figure belongs in the inventory rather than in a build log.

A harvest that retried is a harvest where the venue rate-limited some requests, and rate limiting is not uniformly distributed across time. It concentrates during high activity, which is exactly the market state most hypotheses care about. A substrate built with retries has holes whose placement correlates with the phenomenon under study, and nothing downstream can detect that.

Reporting the retry count alongside the row count is cheap and it is the difference between a substrate you can trust and one you hope is complete.

---

3. Depth against richness

The two venues sit at opposite ends of a trade, and no amount of storage changes which one answers which question.

The identity class in BL-P2-003 exists only because of the second and could not have been produced by the first at any depth. The out-of-sample validation of the direction and carry classes exists only because of the first and could not have been produced by the second at any richness.

That is worth stating because it is the most common substrate mistake: assuming that more data of the kind you already have will answer a question that requires a different kind.

---

4. Where the substrate decided a verdict

Three rows in the Kill Ledger turn on what the substrate could see rather than on what the market did.

The cascade recorder. It captured book and trade snapshots at liquidation events. That is correct for characterising the event and wrong for measuring what follows it, because post-event price paths were only observable where a second event happened to follow. Post-event reversion coverage came out at 29 percent, and the hypothesis was recorded as coverage-limited rather than dead.

The amendment required is scheduled snapshots independent of events. It was specified and not executed before the programme closed.

The wallet scanner. Position state was observed at a 93-second median latency, a deliberate trade between breadth of wallet coverage and freshness on any individual wallet. The research question at that stage was whether identity-level data carries directional information at all, which is a question about a wide cross-section rather than about any one participant. A downstream hypothesis then measured no drift at a 300-second horizon, and the honest reading is that the observation arrived too late to test the claim.

The lead-lag recorder. It captured at one-minute resolution. Price adjustment between large and small instruments turned out to be contemporaneous at that resolution, which closes the question at one minute and says nothing about it below one minute.

All three are recorded as bound to the measurement rather than as falsifications, and BL-D-005 explains why the distinction matters.

---

5. Storage design

Three tiers, each chosen for a different access pattern.

Hot cache. Current state read on the tick path, with time-to-live policies per key pattern and bounded list lengths. Nothing on this tier grows without a bound.

Time-series store. Raw ticks and aggregates, written over a line protocol and read over a query interface, with retention by table class.

Columnar lake. Partitioned files written atomically with explicit gap marking, compacted daily. This is where research data lives, because research reads whole columns across long spans and that is the worst possible access pattern for the first two tiers.

The gap marking matters more than the format. A partition that failed to write is recorded as a gap rather than silently absent, so a downstream reader can distinguish no data from no events.

---

6. The bound that was learned the hard way

An event list in the hot cache was allowed to grow without limit. It exhausted memory and took the production machine down for roughly 38 hours.

The remediation was four-layered and the durable part is one rule: every sink has a bound and a retention policy, declared when the sink is created rather than added after it fills. Raw data moved to the lake and the cache retained only bounded summaries.

R-24 in the data-integrity family and the class K6 in BL-D-006 both come from this incident.

---

7. Retention

Deletion is soft first and hard after a further interval, so that a retention policy misconfigured in the aggressive direction is recoverable.

---

8. What remains, and what does not

No clean held-out data remains on the Bybit substrate. The sealed era was opened once at the Programme II gate and is spent, by design. The earlier sealed era remains unopened but predates 2022 and is not representative of current market structure.

The Hyperliquid recorder is stopped. Its three weeks are what exist.

A new hypothesis on this substrate therefore has three routes: accept that it cannot be certified out of sample and label it accordingly, run a forward shadow and accumulate genuinely unseen data in real time, or move to a different substrate. BL-IS-007 section 6 sets that out.

---

9. Cross-references

BL-EC-005 section 6 for substrate as an access parameter. BL-P2-003 section 3. BL-IS-007 for the seal. BL-D-006 class K6. Evidence E-0038.

---

Copyright (c) 2026 Bijective Labs. All rights reserved.