Summary

  • RIPE NCC attributed the 2 October RRC12 file-production incident to an unusually high volume of BGP messages from one DE-CIX peer, load on a single Kafka partition, and a restart-time rescan that repeatedly timed out.
  • The operator said update-file production at other route collectors was also delayed. It later reported recovery after a configuration fix, while a structural fix for the latent bug remained planned.

The incident started at a route collector and crossed a service boundary. On 2 October, RIPE NCC reported that an unusually high number of BGP messages from one peer at RRC12 had overwhelmed a processing path. The status record then named an effect beyond RRC12: production of update files from other route collectors was delayed too. That is the operational fact worth examining. It is a failure in the production of routing evidence, not a report that the Internet’s forwarding system failed.

RIPE NCC’s incident record says the peer began sending the messages at 01:30 UTC. The processing pipeline consuming from Kafka could not keep up, and a process timed out. When it restarted, the process attempted to scan a large volume of data again, timed out before making progress, and repeated the cycle. The operator later added that a code misconfiguration made the scan take an order of magnitude longer than it should. Correcting that configuration restored progress. RIPE NCC said it would work on a structural fix for the latent bug in the next sprint.

This sequence contains two distinct pressures. First, incoming work exceeded a processing path’s ability to keep pace. Second, recovery itself retried a large scan through a configuration that made the work much slower. The first pressure created a backlog; the second made restarting an ineffective escape from it. The public account does not disclose the peer’s identity, message rate, partition layout, queue depth, timeout values, or exact scan mechanics. It is therefore possible to state the failure sequence without pretending to know its engineering parameters.

The affected output matters because RIS is a measurement service used to inspect routing. Its collector documentation describes physical or virtual machines that collect BGP data through peer sessions; it places RRC12 in Frankfurt at DE-CIX. The MRT documentation says data are stored per collector: dumps represent routing state at a time, while update files record changes over an interval. RIPE documents an eight-hour schedule for dumps and five-minute intervals for updates. Those are service cadences, not proof that each file was complete, timely, or usable during this incident.

At 09:42 CEST on 2 October, the status page said the latest RRC12 update file carried a 01:00 UTC timestamp and its latest bview a 00:00 UTC timestamp. This is a snapshot of the archive’s freshness at one report time. It does not identify when output first stopped, how large the backlog became, or when every downstream user could consume the recovered data. RIPE NCC later said that, after the initial process recovered, RRC12’s bview and update files since 2 October were published successfully. That is evidence of catch-up publication as reported by the service owner, not a byte-level audit of completeness or a measure of user impact.

The distinction between a route and a record of a route is consequential. RFC 6396 defines MRT as an export format for routing information. A missing or delayed MRT file limits what an analyst can observe in that archive at a particular time. It does not, on its own, show that a BGP session ended, a route was withdrawn from the live network, or a customer lost reachability. Conversely, the fact that this was a data pipeline incident does not make the delay trivial: operators, researchers, and incident responders may depend on the archive to reconstruct events.

The status notice did not quantify any such downstream effect, so none should be invented.

The scope of the stall raises a harder design question than whether RRC12 itself needed more capacity. One peer’s burst entered at one collector, yet RIPE NCC reported delays to update files from other collectors. The status report attributes load to a single Kafka partition, which suggests a shared processing dependency in the reported path; it does not publish a full architecture diagram or prove exactly how the work was partitioned. The right conclusion is narrow: the operational impact was not confined to the input location, according to the operator’s own account.

That makes isolation and cross-collector freshness indicators relevant management controls.

There is a revealing planning context, though no stated causal link. RIPE NCC’s Q4 2026 RIS plan was last updated on 17 September, before the incident. It lists improvements to monitoring RIS/RIPEstat data processing as in progress and time-boxed to Q4. It also lists replacing Kafka machines and upgrading to a new Kafka version; the plan says that work had been postponed because of resource limitations. Alternative storage architecture work was also in progress, with the stated goal of reducing HBase dependency. These entries show that processing visibility and platform investment were already recognised workstreams. They do not show that a particular item caused or would have prevented the 2 October incident, nor that it was completed when the files recovered.

The planning page also describes an updated peering approach that aims to retain relevant, diverse peers while keeping data growth sustainable. That objective makes the incident a useful test of a broader operating trade-off. More observed peers can improve the diversity of a route archive, while unusually high input from one peer can increase processing load. The incident does not establish that this peer was unnecessary, abusive, or responsible for the pipeline’s design. A measurement service should preserve a valuable input stream while making its processing cost visible and containing its impact on unrelated collector outputs.

This is where the record should lead next. A useful recovery receipt would identify which outputs were delayed, their last expected and first restored interval, whether the backlog was replayed without gaps or duplicates, and how operators verified the recovered files. A useful capacity test would state the input envelope exercised, the maximum processing lag, restart behavior under backlog, and the point at which another collector’s freshness could be affected. Such controls would turn a status label from “resolved” into evidence about what recovered and what remains structurally vulnerable.

Heng Lu’s Note 70, on the difference between technical capability and practical realities, informs this distinction between a telemetry artifact and the authority readers may give it. It is an editorial lens only. The incident facts and planned work here come from RIPE NCC’s status record and planning documents; the reliability requirements are analysis, not claims about RIPE NCC’s undisclosed internal design.

The status page marked the incident resolved on 6 October. RIPE NCC had already identified the configuration change that let the stuck process recover, while naming a later structural fix for the latent rescan bug. Those are different completion states. A configuration correction can restore production now; a structural repair and a load test can determine whether the same recovery loop is less likely under a future input burst. The public record supports that distinction and leaves the test evidence to be supplied.

Sources