Summary

  • RIPE NCC's 3 November postmortem says a brief network-link interruption failed its primary load balancer over to a standby. A misconfiguration then directed new Whois update database connections to a read-only replica instead of the writable primary; existing application connections remained on the primary.
  • The registry reports intermittent update failures on 28 October from 11:42 to 19:40 UTC. A later review found the failed reverse-delegation requests contained no changes or would have failed predelegation checks, so no reverse delegation would have changed from those requests.

The new route had the wrong destination

The switch worked in one narrow sense: traffic moved to the standby load balancer. The database role behind it was wrong. In its 3 November postmortem, RIPE NCC says the new balancer sent freshly opened Whois update connections to a read-only database. Existing application connections never left the primary and were unaffected. That split, rather than a mysterious on-and-off application, explains why some updates failed while others could continue.

This was a write-path error. A read-only database is useful for serving queries and keeping load off a primary. It is a dead end for a change to a registry object. A standby balancer that routes a new write there can look alive from a network perspective while failing the task an operator actually sent. The old connections mask the mistake: their continued success makes the system appear partially healthy, while every affected new connection is pointed at the wrong role.

The incident's reported failure window ran from 28 October 11:42 to 19:40 UTC. RIPE NCC's original status entry described updates that were not processed and flagged an effect on reverse DNS objects. The later postmortem adds the missing physical mechanism: a short network-link interruption on the primary balancer, failover to standby, then a configuration error in the selected path. It does not publish the configuration key, a connection trace or a count of failed Whois updates. Those records would measure the actual reach of the fault.

The reverse-delegation statement needs its own line. RIPE NCC's 30 October review says none of the failed requests in that group would have changed a delegation. Some submitted no change; the others failed zonemaster predelegation checks. The write path was broken for those requests, but the published evidence gives no lost valid reverse-delegation change. Treating an attempted write as a lost production zone would turn an operations failure into a fact RIPE's own review rejects.

Why an acknowledgement matters more than a green balancer

RIPE NCC's object-processing guide describes updates as object-by-object work, followed by an acknowledgement to the sender. Its acknowledgement guide says a Syncupdates HTTP 200 can accompany individual object failures; the summary and detailed results carry the verdict. For an operator, a connection that stays open or a service indicator that turns green is not the registry receipt. The question is whether the intended object operation succeeded and can be read back from the authoritative record.

The postmortem distinguishes old application connections from new ones; it does not identify every submission method or customer affected. A dated per-method failure count and a trace through the standby path would settle that scope. The central mechanism is already plain: the new road led to the database that could not write.