Summary

  • RIPE NCC says a high-volume BGP peer loaded one Kafka partition; after a process restarted, a latent bug repeatedly re-scanned a large data volume and timed out.
  • A configuration correction allowed recovery, and RIPE NCC says BVIEW and update files dating from 2 October were produced successfully. A structural fix for the restart bug was still planned for a later sprint.
  • Restoring public files is not the same claim as removing the failure path. The incident was about RIPE RIS data production, not evidence that routes or customer traffic disappeared.

The archive came back before the repair was finished

On 6 October, RIPE NCC closed its incident for RRC12 MRT updates and BVIEW files with a specific distinction inside the resolution note. The files since 2 October had been produced successfully after the process recovered. The immediate recovery followed a configuration correction. The organization also said it would work on a structural fix for the latent bug in the next sprint.

Those statements describe two states, not one. The public record was replenished; the mechanism that made a restart fail repeatedly was not yet described as structurally changed. A service can be back in operation while an engineering task remains open. Treating the first fact as proof of the second would make the status page say more than it does.

The sequence began at 01:30 UTC on 2 October, when a DE-CIX peer started sending an unusually high number of BGP messages to RRC12, RIPE NCC said. Its processing pipeline, which consumes from Kafka, did not keep up. A process timed out, restarted, and timed out again before it could make enough progress. RIPE NCC then observed delays in production of other RRC update files as well.

The later explanation added the part the initial notice could not: the incoming volume loaded a single Kafka partition. When its process restarted, a latent bug caused it to re-scan a large amount of data during startup. That scan repeatedly timed out. A code configuration made it take an order of magnitude longer than it should have. Fixing that configuration allowed the process to recover; the structural change was still to come.

What the files represent

RRC12 is RIPE’s Frankfurt route collector at DE-CIX. RIPE’s documentation distinguishes two public artifacts. A BVIEW file is a snapshot of routing state at a point in time; an update file records changes over an interval. The documented cadence is typically eight hours for dumps and five minutes for updates. RIPE says MRT files can support BGP debugging, post-mortems and long-term trend analysis.

That makes a publishing gap consequential for people trying to reconstruct a period from public evidence. It does not make it a routing outage. The incident record establishes delayed file generation, not missing routes, failed traffic, peer intent or customer harm. It does not name the peer or quantify the message rate, backlog, duration of each other collector’s delay or number of analyses affected.

The October archive index now lists BVIEW and update filenames through 7 October. That is a useful present-day check that the public directory contains later artifacts. It cannot independently prove every file is complete, that regenerated data passed a separate integrity test, or that a downstream analysis reached the same result it would have reached with a live stream.

Recovery has more than one finish line

The incident’s stages should not be collapsed into a single green status. First, the input surge stopped being a blocker. Then the restarted process made progress. Then the missing files were generated. Separately, the latent startup path remained scheduled for structural work. These are different operational assertions and should be reported separately.

The distinction is familiar in any measurement system: restoring the record answers “are the files being produced again?” It does not fully answer “has the condition that made recovery fragile been removed?” Nor does it answer “was the recovered series checked for completeness?” The public update answers the first question and names a future task for the second. It does not publish an independent completeness receipt for the third.

That is not evidence that RIPE’s recovery was inadequate. The available status notice is simply narrower than a full post-incident report. It identifies the trigger, a single-partition load, the restart rescan, the configuration correction and the next-sprint structural work. It leaves the peer, rates, queue measurements, per-collector delay window, validation method and release identifier undisclosed.

For users of a public measurement archive, a useful closure record would join those pieces without exposing sensitive implementation details: affected collectors and time intervals; whether files were regenerated; the validation performed; which immediate workaround is active; what structural change remains; and when that change is verified. A date-stamped record would let a later analyst distinguish restored availability from a completed reliability fix.

The careful conclusion is modest. RRC12’s files were restored after a workload surge exposed a restart weakness amplified by configuration. RIPE NCC said structural remediation remained ahead. The archive is not the Internet’s routing table in full, and the outage notice is not evidence of a route failure. But the difference between “the files are back” and “the recovery path is fixed” is exactly the kind of distinction a measurement service should make legible.

Sources