Summary

  • On 8 November, RIPE NCC traced unavailable RIPEstat APIs to a RIS Looking Glass database that exhausted file descriptors under a 32k configured limit.
  • The database and API path recovered, but at 20:06 UTC about one tenth of RRC25 peers were still behind in the Looking Glass replay. The incident remained in monitoring.

A green API is a poor clock. RIPE NCC said its RIPEstat throughput was close to normal by 13:39 UTC on Sunday, 8 November. At 20:06 UTC, it was still reporting delayed Looking Glass data from roughly one tenth of the peers on collector RRC25. The public door had reopened while a corner of the route record was still coming through it late.

The fault began in a very ordinary place: a configured file-descriptor limit of 32k for the database used by the RIS Looking Glass. RIPE NCC said that database ran out of descriptors. An internal API then consumed all available processing slots while handling Looking Glass requests. The first ticket reached RIPE NCC at 00:58 UTC; its public investigation notice at 12:51 UTC named Looking Glass, RIS Peers and other RIPEstat APIs as unavailable. The ticket gives the first reported symptom, not a measured outage start.

At 13:39 UTC, the registry moved the incident to monitoring. The database had recovered, RIPEstat’s API throughput was nearly normal, and latency was improving. The Looking Glass database still had hours of BGP messages to replay. The evening update narrowed the remaining problem to around a tenth of RRC25’s peers and attributed the delay to message volume. RIPE NCC estimated recovery around 05:00 UTC on 9 November; its incident page had not recorded a resolution at this check.

That distinction matters to anyone treating a route view as evidence. RIPE NCC’s Looking Glass documentation says results are organized by collector and peer and include timestamps for the last observed and last confirmed route. A successful request says the API can answer. It does not, by itself, say every peer’s answer is current. The incident turned that familiar caveat into a specific operational split: descriptor capacity, request capacity and replay freshness each recovered on a different clock.

The incident page has yet to provide a final RRC25 peer-freshness receipt. That record, alongside a resolved notice, would close the last part of the story.

Original records: RIPE NCC incident; Looking Glass documentation; RIS Peers documentation.