When a Connected Feed Records Nothing: A Postmortem

While assembling a coverage receipt for our crypto data — an exact statement of what we can and cannot evidence, per venue, before publishing anything derived from it — one venue stood out by three orders of magnitude in the wrong direction.

Over the 24 hours ending 2026-09-02 00:00 UTC, our order-book collection stored 154 snapshots from Kraken, covering 12 of 1,440 minutes. Every other connected venue stored between 29,720 and 1,326,344 snapshots over the same window, with data in every one of those minutes.

The connection was not down. It was established, it was receiving messages from the venue, nothing in the process was throwing errors, and no alert fired at any point. By every liveness check we ran, the feed was healthy. It was storing 0.83% of the minutes it was supposed to cover.

The venue's WebSocket delivers order-book updates as incremental deltas — a message might contain only bid changes, or only ask changes, because only one side moved. This is normal and well documented; reconstructing a book from deltas is the consumer's job.

Our connector for that venue forwarded those raw deltas onward as though each were a complete book payload. The component that converts an incoming payload into a storable snapshot requires both sides to be present and returns nothing when either side is empty. So it returned nothing, on almost every message, and it did so at debug log level — which is to say, silently.

The 154 snapshots that survived were the full order-book images the venue sends on subscribe. That is why the surviving coverage arrived in whole-pair batches roughly seventeen times per 48 hours: those were reconnects. Every reconnect produced one usable snapshot per pair, and then the feed went quiet again while appearing entirely healthy.

The connector for another venue in the same fleet already maintained a local book and emitted complete snapshots. This one did not. The difference was invisible from outside the process, because both were connected, both were consuming messages, and neither was erroring.

Every monitor we had was watching the wrong layer. Connection state was up. Message throughput from the venue was normal. Process health was fine. The loss happened after ingestion and before storage, in a conversion step whose failure path was a silent return rather than an exception.

This is the general shape of the problem, and it is not specific to us. Independent accounts of real-time crypto data describe exactly this class of failure: connections that stay open but stop delivering useful updates, missing depth for particular venues, and reconstructed books that do not cover full days. Liveness and data completeness are different properties, and monitoring the first says very little about the second.

The check we were not running is per-venue write rate compared against peer venues. A venue writing three orders of magnitude fewer rows than its peers is anomalous regardless of what any health endpoint reports, and catching it needs no knowledge of the failure mode.

The connector now maintains local book state and emits complete two-sided snapshots, matching the behaviour of the connector that was already working.

Acceptance was measured against the live venue before and after. Before: 3 of 2,897 messages survived conversion into a storable snapshot — 0.10%. After: 1,487 of 1,487 — 100%. In the five minutes following deploy, the venue recorded 1,893 snapshots across 14 pairs, which placed it third among all connected venues by write rate.

The fix landed at 2026-09-02 01:03 UTC.

Our 16-venue feed latency study covers the window during which this venue was broken, and excludes it from the comparison for exactly this reason. Treating a 154-snapshot sample as comparable to a million-snapshot one would have been misleading, so the exclusion is stated in the study rather than performed quietly.

Any measurement recomputed after 2026-09-02 01:03 UTC should include the venue, and must not reuse figures from the earlier window.

If you consume exchange market data — from venue WebSockets directly or through a vendor — this failure mode is worth ruling out explicitly, because it does not announce itself. The checks below are ordered by how much they catch relative to how hard they are to add. The first one would have caught ours on the day it started.

• ▶Alert on per-venue write rate against peer venues, not on connection state — relative anomalies need no knowledge of the failure mode
• ▶Never let a conversion or validation step fail silently; a dropped message is an event worth counting even when it is expected
• ▶Track sequence numbers where the venue provides them, and count detected gaps as a first-class metric
• ▶Measure minute-level coverage, not uptime — the useful question is what fraction of minutes hold data, not whether a socket was open
• ▶Distinguish a broken feed from a quiet market; both produce gaps, and only one is your problem to fix
• ▶Reconcile stored data against a live sample of the venue periodically, which is how the acceptance figures above were obtained

A data provider's error rate is not usually public. We publish coverage limits, a corrections log and postmortems like this one because a dataset with an undisclosed hole produces confident findings that are wrong in a way no reader can detect. Stating the hole is less impressive and considerably more useful.

The windows behind every figure here are stated so they can be checked. Raw order-book data is retained for five days, so the before-and-after counts are recomputable from our storage only within that window; after it, this article and the coverage receipt behind it are the record.