Ed Shryane traces RIPE’s 60-hour reverse-DNS delay

Summary

RIPE NCC says an NRTM failure lasted from 00:06 UTC on 2 August to 12:30 UTC on 4 August. Reverse-DNS updates waited during that 60-hour-24-minute window, then the pending queue was processed.
At RIPE 93, database engineer Ed Shryane traced the failure to escaped characters being decoded into an extra newline inside an RPSL object. The repair preserved the escapes and checked the stream’s recent objects and closing footer.

Analysis

The domain record could be changed in the RIPE Database while the corresponding update still failed to reach global DNS. The broken handoff sat in RIPE NCC’s Near Real Time Mirroring service: the reverse-DNS provisioning system uses NRTM to pick up database changes.

RIPE NCC’s 6 August note to the Database Working Group dates the affected period precisely: 00:06 UTC on 2 August through 12:30 UTC on 4 August. That is 60 hours and 24 minutes. The note says reverse-DNS updates were delayed over that span and all pending updates were processed after a fix. RIPE’s status record labels the incident major; its recovery updates say pending domain changes had been published and Reverse DNS was operational again.

Shryane described the technical cause at the RIPE 93 Database session. A decoding step unescaped characters in an RPSL object and inserted an additional newline. The resulting object was syntactically invalid, yet RIPE NCC had accepted it. RPSL is a text format with structure carried by its lines; RFC 2622 defines that syntax. A harmless-looking conversion had changed the record the mirror stream had to parse.

The fix had two parts. The team stopped unescaping the characters and left them as they were. It then went back over the last 1,000 objects in the stream, checked that they parsed correctly, and verified a consistent footer at the end of the NRTM response. That last check matters: consumers need a readable stream and a clear point at which the response ends.

The operational chain is short and unforgiving: database object, mirror stream, reverse-DNS provisioning, published DNS update. A text conversion inside the registry became a delay at the far end of that chain. RIPE’s public material gives the duration and recovery, but no count of affected mirror clients or domains; those counts would change the estimate of the external blast radius.