BL-IN-005: The Dead Feed

Status: RELEASED

Classification: PUBLIC

Revision: 1

Date: 2026-07-16

Incident date: 2026-07-16 Class: K2, verified once and assumed verified forever Rule produced: R-18, and the requirement that subscribe failures are logged with the venue's own message

1. What was believed

The liquidation stream was believed connected and working. The brain log showed the stream state as connected, with periodic reconnects, and that was read as evidence of a healthy subscription.

An entire strategy candidate depended on that feed. A cascade-exhaustion hypothesis needed liquidation events to detect the cascades it was built to trade, and a dedicated recorder had been deployed to accumulate them.

2. The trigger

The recorder had been live for 42 hours and had written zero files.

Not corrupt files, not empty files. None. The output directory had never been created.

3. The investigation

Six steps, all on 2026-07-16.

Confirm the recorder's output directory had never been created; its log contained only a start line.

Check the cache the recorder polls. The liquidation key did not exist. Wildcard searches for any liquidation-shaped key returned nothing, while the trade, order book and funding streams were alive across 583 keys. Only the liquidation substrate was dead.

Check the brain logs. The stream showed as connected, with periodic reconnects, and had produced zero cascade detections across its entire lifetime of more than nine days.

Probe the venue websocket directly. Subscribing to liquidation.BTCUSDT returned:

subscribe_success = 0
ret_msg = "error:handler not found,topic:liquidation.BTCUSDT"

Subscribing to allLiquidation.BTCUSDT, along with eleven other instruments, succeeded.

Cross-reference the venue's own documentation for the replacement topic's schema.

Run two live observation probes, of roughly fifteen and twelve minutes, to resolve a semantic question the schema change introduced.

4. The finding

The venue had deprecated liquidation.{symbol} in favour of allLiquidation.{symbol}. The system had been subscribed to a dead endpoint for more than nine days, receiving nothing.

Two things made it invisible, and both are more interesting than the topic rename.

The failure was reported and then discarded. The venue answered the subscription with an explicit error. The client checked whether a success field was present and continued past anything else without logging:

if m.success.is_some() { continue }

A message carrying a failure has a success field. It is present, and it is false. The guard matched, the message was dropped, and a total-silence failure produced no error anywhere in the system.

Zero is a plausible reading. Liquidation cascades are rare by nature. A feed delivering nothing looks exactly like a quiet market, and there was no baseline of expected event rate that would have distinguished the two.

That combination, a swallowed error and an output whose absence is unremarkable, is the general shape of this failure class. It is worth looking for wherever a system consumes an event stream that is legitimately sparse.

5. The second defect, found because of the first

The replacement topic changed a field's meaning.

The old topic reported the side of the order that performed the liquidation. The new topic reports the side of the position being liquidated. The mapping inverts:

old:   is_long = (side == "Sell")      order-side convention
new:   is_long = (side == "Buy")       position-side convention

Left uncorrected, this would have flipped the direction of every cascade event the strategy consumed, and the strategy would have run backwards on a feed that was finally working.

The verification is worth reading because of what it declines to claim. During a synchronised sell-off across three major instruments, 25 observed liquidation events came back 23 to 2 in favour of one side, consistent with long positions being liquidated in a falling market. That macro pattern, together with the venue documentation, was accepted as evidence.

A per-event cross-check against the aggressor tape was also run and returned 10 against 11 across 21 events, which is indistinguishable from a coin flip. That check was reported as inconclusive and was not used as supporting evidence. The confidence in the fix was recorded as high but not absolute, pending observation of an upward move.

Reporting a test that failed to discriminate, and then declining to lean on it, is the standard this record is trying to demonstrate.

6. What it cost

Nine or more days during which an entire research line could not proceed.

The programme had been carrying a ledger entry assuming two to four weeks of accumulated cascade data existed. That entry was reset to zero. Accumulation restarted only after the fix deployed, which pushed the hypothesis's verdict past the programme's runway deadline. It appears in the Kill Ledger as B1, status OPEN, and it is still open for that reason.

No capital was at risk.

7. The fix

Four changes, test-driven, with four new tests added:

Subscribe to the replacement topic.

Add a parser for the new message shape, which delivers an array rather than a single object, skipping malformed elements without failing the batch.

Invert the position-side mapping.

Add the venue's own message field to the deserialised message type, and replace the swallowing guard with an explicit error log when a subscription fails.

Verification before deploy: 320 of 320 unit tests in the compiled path, 378 of 378 in the Python path. A grep against the compiled binary confirmed the new topic present twice and the old topic absent.

Verification after deploy, at 15:04 UTC: the container stable with no crash loops and no rejected subscriptions in the log; the liquidation key populated within five minutes with four real events across four instruments; and the recorder wrote its first files ever, three flushes each carrying one liquidation event, one order book snapshot and 500 trades, after 42 hours of nothing.

8. The rule

Two rules came out of this.

R-18 requires that every input is cross-checked against a second source before a verdict is issued. A single feed reporting itself as connected is not evidence that it is delivering.

The second is narrower and belongs in any ingest client: a subscription failure is logged with the venue's own message text. The system had the error in hand and threw it away.

There is a third, harder lesson that resists being written as a rule. A sparse event stream needs an expected rate, and a monitor that fires when observed falls far enough below it. Without that, silence is indistinguishable from absence, and no amount of connection-state logging closes the gap.

9. What the source does not settle

The onset of the blind window is not pinned to a calendar date. The record gives "more than nine days" for the brain and 42 hours for the recorder, but not when the venue deprecated the topic relative to when the system was deployed against it.

The position-side fix is recorded as high confidence but not absolute, pending observation during an upward move, and the record does not report whether that confirming observation was later made.

Batch-subscription safety, the risk that one delisted instrument in a group fails the whole connection, is flagged as unimplemented with no resolution recorded.

10. Cross-references

BL-D-006 class K2 and section 5. BL-D-009 rules R-18 and R-21. Ledger row B1.

Copyright (c) 2026 Bijective Labs. All rights reserved.