Summary

  • A feature that created one Kafka producer per API request drove nearly 4.2 million additional producers per hour at peak, exhausting broker heap and degrading event, notification, integration, API, mobile, and status-communication paths.
  • The first response restored service without identifying and removing the trigger; a same-day recurrence connected abnormal Kafka traffic to the feature and led PagerDuty to roll back the offending code.

An incident-management platform occupies an unusual position in the chain of operational responsibility. Its customers do not use it only when conditions are normal. They depend on it at the moment another system is failing, when a delayed event, a missing notification, or an unreliable status update can distort the response to a separate emergency. That does not make uninterrupted service possible. It does make the allocation of control unusually important. The relevant question is not whether PagerDuty could promise that Kafka would never fail.

It is whether the company controlled the decisions that created the failure, the signals that delayed diagnosis, the mechanisms that widened impact, and the evidence needed to show that the same path had been closed.

PagerDuty's account of its August 28, 2025 outages provides a concentrated case. A feature intended to support later analysis of API-key usage caused a new Kafka producer to be created for every API request. At peak, PagerDuty says, Kafka was tracking nearly 4.2 million additional producers per hour, 84 times its typical number of new producers. The metadata burden increased memory pressure on Kafka brokers, drove Java Virtual Machine garbage collection into thrashing, exhausted heap memory, and cascaded through a cluster that supported asynchronous work across the service. The visible result was not one clean outage.

It was a mixture of rejected incoming events, delayed processing, delayed notifications, API errors, duplicate or delayed webhooks, impaired integrations, and status updates that responders drafted but customers could not see.

The chronology matters because the same technical condition appeared twice on the same day. The first incident began at 03:53 UTC. PagerDuty stabilized Kafka, recovered dependent services, processed stale work, and reported normal operations by 10:10 UTC. The company did not identify and remove the feature trigger during that response. A smaller recurrence began at 16:38 UTC. Responders repeated earlier mitigation, found the abnormal traffic pattern, rolled back the offending code, and mitigated customer impact in about 50 minutes. PagerDuty says every service was fully restored by 20:24 UTC.

The first recovery therefore restored service without conclusively removing the initiating condition. The second response connected the infrastructure symptoms to the software change.

That sequence turns a software defect into an accountability record. Root cause, triggering event, contributing conditions, detection failure, response failure, and recovery failure were related, but they were not identical. Treating them as one undifferentiated “Kafka outage” would obscure who controlled each layer. It would also misstate the public evidence. PagerDuty did not identify Kafka as a defective third-party product. It identified a logical error in its own feature code and the way that code interacted with the company's architecture.

The distinction is central: responsibility should follow control over the mechanism, not the most recognizable technology name in the stack.

The Evidence Is Detailed, but It Is Still a Company Account

The factual foundation for this reconstruction is PagerDuty's engineering postmortem, published on September 5, 2025. It is a primary operational record from the organization that ran the affected system. It gives start and restoration times, the feature mechanism, the first response's diagnostic path, the reason no rollback occurred during that response, the recurrence, and numerous impact measurements. Those details support a tighter analysis than a collection of social-media complaints or an unsourced outage summary would permit.

The same source has an unavoidable limit. A company postmortem is not an independent audit. It can confirm what PagerDuty publicly represented, but it cannot by itself establish every customer's experience, every internal decision, or every downstream loss. The postmortem says PagerDuty retained previously accepted events and data. That statement does not mean every attempted event was accepted: PagerDuty separately says some incoming events may have been rejected, including 502 responses from the Events API at peak impact. It also does not mean that a delayed notification caused no harm.

It means the narrower proposition the company made about data already accepted by the platform.

The evidence falls into four classes. Confirmed facts are statements in the postmortem, attributed to PagerDuty where attribution matters. Evidence-backed inferences connect those facts to control responsibilities, but do not pretend to reveal undocumented intent. There is no necessary disputed allegation at the center of this case; the central causal account comes from PagerDuty itself. Unknowns remain unknown: the source does not provide a customer-by-customer harm ledger, prove that every customer or region was affected, quantify independent business losses, or disclose the complete internal approval and test record for the feature.

Those boundaries prevent severity from becoming speculation.

Before 03:53 UTC: A Reporting Feature Entered a Critical Path

PagerDuty describes Kafka as the backbone of its asynchronous architecture. That description establishes the first important control fact. Kafka was not peripheral to the service. Work dependent on it included the processing paths that connected incoming events to notifications, integrations, webhooks, chat systems, mobile actions, and other capabilities. A change that increased load on that backbone therefore carried a different potential blast radius from a change isolated to a noncritical reporting interface.

The new feature was meant to support analysis and reporting on API-key usage trends. PagerDuty expected its volume to be roughly comparable to API-key use. On its face, that is a reasonable product objective: record usage information, send it to a Kafka topic, and analyze it later without holding up the initiating request. The accountability problem did not arise from the objective. It arose from the implementation's unit of work. Instead of reusing a producer to publish messages, the feature instantiated a new Kafka producer for each API request.

That distinction is easy to compress into a sentence and easy to underestimate. A producer was not merely a transient local entity with no effect beyond the request that created it. Kafka had to track metadata associated with each producer. Repeating that operation at API-request scale transformed application traffic into control and memory pressure inside the message-queue cluster. PagerDuty's measured peak—nearly 4.2 million additional producers an hour—was 84 times the ordinary rate of new producers. The feature therefore did not simply add the expected stream of usage messages.

It changed the population of producer identities the brokers had to manage.

The confirmed fact is the logical error and the resulting producer volume. An evidence-backed inference follows: assurance focused only on the number of messages would have measured the wrong risk. A test could observe that each API request generated one expected usage event and still miss the multiplying metadata cost created by one producer per request. Similarly, a gradual rollout could limit immediate traffic yet fail to expose the causal relationship if reviewers watched application success rates without watching producer creation, broker heap, and garbage-collection behavior as linked indicators.

The postmortem discloses a staged rollout from 1 percent on August 21 to 5 percent and then 25 percent on August 27, followed by 75 percent on August 28; it does not disclose the exact preproduction test suite, approval chain, or alert thresholds. It would be unsupported to say that no testing occurred or that a named employee ignored a known danger. The narrower conclusion is defensible: whatever controls existed did not stop a per-request producer pattern from reaching a shared Kafka backbone, and the initial monitoring picture did not promptly identify that pattern as the source of systemic memory pressure.

That is a contributing condition rather than the triggering event itself. The code created the capacity for abnormal producer growth. Live API traffic exercised it. Kafka accumulated metadata. Broker memory pressure rose. Garbage collection began consuming effort without restoring stable headroom. Heap exhaustion then moved the problem from inefficient behavior to service failure. Each step had a different possible signal. Accountability depends in part on whether those signals were visible together or separated across application and infrastructure teams.

03:53 UTC: The First Failure Looked Smaller Than It Was

PagerDuty marks 03:53 UTC as the beginning of the first incident. A failure on one of its Kafka message-queuing systems triggered cascading issues that disrupted or delayed processing of new incoming events for some customers in the US Service Regions. “Some customers” and “US Service Regions” are material limits. The record does not support a claim that every PagerDuty customer worldwide lost service. It supports a serious, multi-capability degradation within the scope the company reported.

The early alerts suggested a failure of one broker and possible hardware trouble. That diagnosis was plausible enough to direct the first response. Engineers expanded the cluster and removed a broker. Those actions addressed the apparent failed component and increased capacity. They did not remove the feature behavior that was continuously generating new producers. When additional brokers ran out of memory, the incident ceased to look like a single-node problem. PagerDuty then recognized a systemic software issue.

This is the detection failure in its precise form. It was not a failure to notice that something was wrong; alerts did fire, and responders acted. It was a failure of causal detection. The initial telemetry presented a local infrastructure symptom more clearly than it presented the application-level behavior that was driving the cluster toward exhaustion. A detection system can be fast in announcing pain and still be slow in identifying its source. For a shared asynchronous backbone, that difference determines whether the first intervention removes a failed component or stops a workload that will exhaust the replacement capacity as well.

Calling the early interpretation unreasonable would go beyond the evidence. Hardware faults and broker failures are ordinary hypotheses when one broker alerts. The accountability question is whether the observability design made a systemic alternative available soon enough. Producer creation rising to 84 times the typical new-producer rate was a distinctive signal. Broker heap consumption and garbage-collection thrash were related signals. A feature rollout was a relevant change signal. The source does not say when each became visible to responders or whether one console correlated them.

It does show that the relationship was not established before multiple brokers suffered memory exhaustion.

That lag expanded the effective blast radius. Cluster expansion added room, but the abnormal creation pattern could consume additional room. Removing one broker eliminated a symptomatic node, but not the stream of metadata work. The response became effective only after the team treated the event as systemic: PagerDuty doubled heap size, rolled the cluster, stabilized Kafka, and then restored services that depended on Kafka. Those steps addressed immediate resource pressure and allowed queued work to move again.

The root cause was not limited public evidence heap in the ordinary sense. More heap was a mitigation. If the same erroneous producer pattern remained active, capacity could defer exhaustion without making the design sound. Nor was the root cause the backlog that appeared during restoration. The backlog was a consequence of impaired processing and a source of later recovery load. Precise classification matters because otherwise an organization can document a successful capacity intervention while leaving the initiating software behavior in place.

The Impact Was a Chain of Unequal Failures

An incident platform has at least two relevant flows. It must receive and process signals, and it must turn those signals into useful action through notifications and integrations. PagerDuty's first outage affected both. At peak, some incoming events may have been rejected with 502 responses from the Events API. Other events were accepted but delayed. Outbound notifications, webhooks, chat integrations including Slack and Microsoft Teams, REST API operations, mobile acknowledgement and resolution, and several enterprise integrations were impaired in different proportions and for different durations.

The difference between rejection and delay is operationally important. A rejected event can require the sending system to retry or can disappear if that system lacks a reliable retry path. A delayed event remains in a queue, but may arrive after the time when it would have changed a response decision. A duplicated webhook or chat message can make responders wonder whether there are multiple incidents or whether a state transition happened twice. An API error may prevent acknowledgement or updating even while a notification has already reached a person. A single “availability” percentage would conceal these different burdens.

PagerDuty states that around 14 percent of events were delayed. Event rejection peaked at about 95 percent for 38 minutes. Those figures should not be merged into a claim that 95 percent of all events were permanently lost. They measure different conditions. The company also reports that about 16 percent of email-ingestion events were delayed and less than 1 percent were not processed. For change events, 6.5 percent were rejected for 55 minutes. Each number is bounded by the category and duration in the postmortem.

On the outward path, about 23 percent of notifications were delayed by at least five minutes for 209 minutes. PagerDuty says no notifications were dropped completely. That statement is reassuring in one dimension and serious in another. A notification system can ultimately deliver every message and still fail a time-sensitive purpose. Five minutes is not an abstract delay when the message is intended to mobilize a response. The public record does not establish what happened in any individual customer's incident, so it cannot support an invented emergency or financial loss.

It does establish a prolonged period in which almost a quarter of notifications, by PagerDuty's measurement, crossed a five-minute delay threshold.

The REST API saw increased error rates for about 150 minutes. PagerDuty reports that 18.87 percent of create requests returned 5xx responses for 130 minutes and 4.35 percent of update requests returned 5xx responses for 190 minutes. Mobile workflows were also affected: 6.06 percent of users were unable to acknowledge or resolve incidents through the mobile application. These failures can interact. If a notification is delayed, an acknowledgement request fails, and an integration retries later, the customer's record of what responders knew and when can become less reliable even if the underlying incident data is eventually preserved.

PagerDuty also lists delayed or dropped integration events for Jira, ServiceNow, Salesforce, and Zendesk for about 100 minutes. Webhooks were delayed, dropped, or duplicated for about 100 minutes. Chat systems experienced delays and duplicate messages. These are not interchangeable conveniences. Customers often embed such outputs in ticket creation, escalation, ownership, and audit trails. PagerDuty's source does not measure the quality of each customer's downstream reconciliation.

It confirms that the platform handed those customers recovery work: retrying, checking for gaps, suppressing duplicates, and determining whether a late message reflected current state.

The evidence therefore supports a bounded inference. The outage's cost was not confined to time when a webpage was unreachable. It transferred uncertainty into customer operations. The more a customer had automated around PagerDuty outputs, the more it needed to distinguish accepted from rejected events, delayed from current notifications, and duplicates from new state. That inference does not quantify a loss. It identifies where responsibility for reliable semantics matters in a dependency sold for incident coordination.

A Status Page Failed at the Moment Status Mattered

The first incident also disrupted PagerDuty's external communication process. The company says updates were drafted internally but did not appear publicly on the status page. Additional engineering teams were engaged, and responders used backup procedures to add updates manually. PagerDuty records delayed status-page updates for about 100 minutes.

This was a response failure separate from the Kafka root cause. The feature bug did not logically require public communications to be delayed. The delay arose because the process that converted internal incident knowledge into external status information did not complete successfully, and the backup path required manual intervention. A status page is supposed to reduce customer uncertainty when the primary service is degraded. If publication depends on a path coupled to the affected environment—or on an external process that is not independently verified—customers can lose both service and authoritative explanation together.

The postmortem does not provide the internal drafts, the exact publication dependency, the time of each attempted update, or the full manual procedure. It would be speculation to claim a particular tool failed or a particular team neglected a duty. The confirmed boundary is narrower: internal updates existed, public display did not occur on time, more engineers were engaged, and backup procedures were used. The evidence-backed lesson is that communication recovery deserves the same rehearsal as technical recovery. A drafted message has no operational value to a customer who cannot see it.

The delay also complicates customer diagnosis. When an incident-management platform behaves inconsistently, customers must determine whether their own monitored systems stopped emitting events, whether an integration failed, or whether the platform is delayed. A timely independent status signal can narrow that search. A missing update pushes each customer toward local tests, support contacts, or conjecture. That duplication of diagnostic work is a predictable consequence of communication failure, even though the postmortem does not quantify it.

Stabilization at 10:10 UTC Was Not Elimination of the Trigger

PagerDuty says all services and system capabilities returned to normal operations by 10:10 UTC. Reaching that point required stabilizing Kafka and restoring the services that depended on it. As processing resumed, customers could receive delayed notifications and alerts while PagerDuty worked through stale messages. Some could receive duplicate webhooks. Recovery therefore had a queueing tail: infrastructure could be stable while old work continued to emerge into customer systems.

That tail is a recovery condition, not proof of a new failure. Queued messages have to be processed or deliberately discarded under an explicit policy. PagerDuty says previously accepted events and data were not lost, so processing the backlog was consistent with preservation. But preservation creates ordering and timeliness questions. A customer receiving an old alert needs enough context to know it is old. An automated endpoint receiving a retry or duplicate needs idempotent behavior or a reconciliation process. The platform and the customer each control different parts of that boundary.

The more consequential limitation was that the offending feature had not been rolled back. PagerDuty explicitly explains why: during the first incident, the trigger was not evident, and responders concentrated on stabilizing Kafka. That statement should be taken seriously rather than rewritten as intent or indifference. Incident responders often must choose between restoring a critical shared system and investigating every possible upstream cause. Immediate stabilization can be the rational priority when the causal picture is incomplete.

Yet the decision has an accountability consequence even if it was reasonable. The service was operational, but the initiating code path remained capable of producing the same abnormal load. The first response had increased heap and rolled the cluster; it had not conclusively demonstrated that the condition which consumed the heap was gone. In reliability terms, recovery had been achieved at the service level but not yet at the causal level. That gap became visible later the same day.

This is the first recovery failure, carefully defined. It does not mean the restoration work failed to restore service at 10:10. It means recovery assurance did not establish that the trigger had been removed before the incident was considered operationally normal. The public source does not say what monitoring or change-freeze conditions were applied during the interval. It does show that the same class of Kafka issue recurred at 16:38.

16:38 UTC: Recurrence Turned Mitigation Into Diagnosis

PagerDuty describes the second incident as a smaller recurrence. Responders reapplied the mitigation steps used earlier and limited customer impact within about 50 minutes. This faster containment suggests the team had learned how to stabilize the affected system. It does not by itself show that the earlier root cause was understood. The decisive change in the second response was the discovery of the abnormal traffic source and the rollback of the offending code.

The recurrence sharpened the evidence. A broker or hardware explanation could no longer account comfortably for the whole day. The same platform experienced a related Kafka failure after the first cluster intervention. Responders now had a recent baseline, known mitigation steps, and a smaller search window around changes and traffic behavior. PagerDuty says it discovered the source of the abnormal traffic patterns during this response. Once the feature code was linked to the producer surge, rollback addressed the initiating software condition rather than only its memory consequences.

The distinction between “impact mitigated” and “fully restored” is again important. Customer impact from the second event was mitigated in roughly 50 minutes, according to the company. PagerDuty reports full restoration of all services at 20:24 UTC. Those statements can coexist. An incident can stop causing new acute harm while dependent systems, queues, integrations, and verification tasks continue toward normal state. Compressing the record to a 50-minute outage would omit that recovery interval. Calling the entire interval equally severe would also be imprecise.

Rollback supplied stronger causal evidence than capacity expansion alone. The postmortem does not provide a controlled experiment, but the sequence supports an evidence-backed inference: abnormal producer traffic stopped being regenerated once the offending feature was removed, allowing the repaired infrastructure state to hold. The postmortem's own causal narrative identifies the logical error, producer multiplication, metadata pressure, garbage-collection thrash, heap exhaustion, and cluster cascade. No competing public cause is presented in the published record.

There is consequently little value in treating the cause as disputed for rhetorical balance. The responsible distinction is between a confirmed company account and independent verification that is not in the public record used here. PagerDuty publicly owned the feature mechanism. The unknowns concern the internal control record and downstream customer outcomes, not an alternative allegation of sabotage, crime, or deliberate conduct. Nothing in the published evidence supports such claims.

The Causal Ledger

The root cause was the feature's logical error: creating a Kafka producer for each API request rather than reusing a producer to publish usage messages. That implementation made ordinary request volume generate extraordinary producer metadata. PagerDuty controlled the feature code and the service architecture in which it ran. This is the deepest documented cause because removing the erroneous behavior addressed the initiating mechanism.

The triggering event was live execution of that code at scale. API requests repeatedly instantiated producers until Kafka was tracking almost 4.2 million additional producers per hour at peak. The trigger was not a malicious traffic event in the public account, and it was not identified as a Kafka product defect. It was the encounter between PagerDuty's feature behavior and production request volume.

The contributing conditions included the criticality of the shared Kafka backbone, the metadata cost of producer proliferation, finite broker heap, and a garbage-collection response that became thrashing rather than relief. The architecture allowed pressure created by one analytical feature to affect multiple customer-facing and response-facing paths. The public record also supports concern about assurance coverage: the implementation reached production without a control preventing or promptly flagging per-request producer creation.

The source does not reveal which specific test or approval should have caught it, so that concern remains an inference about the result, not a claim about a named process violation.

The detection failure was diagnostic. Alerts initially pointed responders toward one broker and possible hardware trouble. The system detected failure symptoms, but the cross-layer correlation among a recent feature, producer-count acceleration, broker heap pressure, and garbage-collection behavior did not produce the right cause early in the first event. PagerDuty did recognize the systemic nature once additional brokers exhausted memory. It found the abnormal traffic source only during the recurrence.

The response failure had two parts. Technically, early actions—adding cluster capacity and removing a broker—treated the apparent local failure without stopping the workload that would consume more capacity. This is not a claim that the actions were irrational; it is a statement that they were incomplete relative to the actual cause. Communicatively, internally drafted status updates did not become timely public updates, and backup procedures required added engineering attention and manual posting.

The recovery failure was also two-layered. The first restoration returned services to normal but left the offending feature active because its role was not yet known. That left recurrence risk. Separately, recovery generated stale-message and duplicate-output effects that customers had to interpret. During the second incident, responders used known mitigations, identified and rolled back the code, then continued restoring all dependent capabilities until 20:24.

The consequence was a portfolio of bounded impacts rather than a single binary outage. Incoming work could be rejected or delayed. Outgoing notifications could be late without being permanently dropped. APIs could return errors. Integrations could delay, drop, or duplicate work. Mobile users could be unable to acknowledge or resolve. Public status information could lag internal knowledge. These effects matter precisely because PagerDuty sits between detection and response for its customers.

This classification prevents two common errors. The first is assigning the entire incident to one coding mistake and ignoring conditions that allowed that mistake to stress a shared backbone and evade early diagnosis. The second is distributing blame so broadly that no controller remains visible. Kafka, customer systems, and network traffic were part of the environment, but PagerDuty's evidence places the decisive feature, architecture, observability, mitigation, communication, and rollback controls within PagerDuty's operational domain.

Accountability Follows the Controls PagerDuty Held

PagerDuty controlled the feature design. It decided how API-usage data would be produced and sent to Kafka. It controlled the code review, test environment, rollout method, and production observability, even though the public source does not disclose the content of each control. It controlled the shared Kafka architecture and the capacity and alerting around it. It controlled the incident response, the status-publication process, the rollback, and the postmortem. Those controls make PagerDuty the primary accountable actor for preventing, detecting, containing, explaining, and repairing this failure.

Primary accountability is not the same as unlimited liability for every downstream event. The postmortem does not quantify customer business losses or show that every delayed message caused damage. It does not establish contractual outcomes. A forensic allocation should not leap from “23 percent of notifications were delayed by at least five minutes” to a total monetary figure. It should ask what evidence PagerDuty can supply to affected customers so they can reconstruct their own timelines: acceptance records, rejection responses, delivery timestamps, retry behavior, duplicate identifiers, and service-region scope.

Customers retained some controls, but not equivalent ones. A customer could monitor its own systems independently, preserve local event queues, implement retry logic for 502 responses, make webhook consumers idempotent, maintain alternative escalation contacts, and avoid treating one vendor's status page as the only source of truth. Those are prudent dependency controls. They do not transfer responsibility for a per-request producer defect to the customer. Customers could mitigate their exposure; they could not inspect or roll back PagerDuty's internal feature.

The same asymmetry applies to notification delay. Customers decide which events enter PagerDuty and how their teams respond. PagerDuty controls whether an accepted event moves through its platform on time. A mature shared-responsibility account should identify both sides without creating false equivalence. The vendor owns internal processing reliability and truthful incident evidence. The customer owns contingency design for the residual possibility that the vendor is impaired, especially when the vendor is part of the customer's own emergency path.

Kafka should not be assigned a product failure on this record. PagerDuty's postmortem describes Kafka's metadata tracking and memory behavior as the environment in which the feature error became destructive. It does not say Kafka violated a documented guarantee or contained a defect that caused the event. Calling this a “Kafka failure” may be operationally convenient, because Kafka brokers exhausted heap, but it is incomplete as an accountability conclusion. The actionable root cause sat in PagerDuty's creation of producers and the absence of an effective constraint on that pattern.

The status-page issue belongs to PagerDuty as well. Customers did not control whether internal drafts appeared publicly. A resilient communication design should not assume that the same conditions impairing service will leave every publication dependency intact. The evidence does not tell us whether PagerDuty's backup process was routinely tested. It tells us that the backup was needed and that updates were delayed for about 100 minutes. Accountability after the event requires more than saying manual posting ultimately worked; it requires evidence that the independent path is fast enough under realistic failure conditions.

There is also an accountability duty in measurement. PagerDuty's postmortem is unusually specific about error percentages and durations. That precision helps affected organizations avoid an all-or-nothing interpretation. It also creates follow-up questions. Were percentages calculated across all relevant requests or a scoped population? Can customers obtain tenant-specific data? How were delayed, dropped, rejected, and duplicated outputs classified? The published summary does not answer those questions.

Raising them is legitimate because measurement controls the boundary between a general incident narrative and a usable customer reconstruction.

Finally, accountability extends from explanation to proof of durable repair. Rolling back the code stopped the documented trigger. It does not, on its own, prove that a similar entity-lifecycle error cannot reach another shared queue or that monitoring will correlate the next application-level anomaly with broker resource pressure. Durable evidence would include a constraint on producer creation, tests that fail on abnormal producer cardinality, alerts tied to rates rather than only broker death, rollout checks linked to infrastructure health, and a communication path verified independently of the primary incident platform.

These are evidence-backed requirements derived from the failure, not claims that PagerDuty has or has not implemented every item.

What the Public Record Cannot Resolve

The postmortem does not identify every affected customer or quantify every region. It says some customers in the US Service Regions experienced disruption or delay. Any broader claim would exceed the record. It does not enumerate individual incidents for which a notification arrived too late, and it does not independently calculate economic loss. Those gaps should not be filled with hypothetical victims presented as fact.

The record also does not expose the feature's complete governance history. We do not know the precise code-review discussion, test cases, load-test shape, deployment stages, on-call handoffs, or decision thresholds used between 03:53 and 10:10. We know the outcome: the feature reached production, the producer count surged, early signals resembled a broker or hardware fault, and the trigger was not identified during the first incident. That is enough to test control design, but not enough to accuse an individual of knowingly accepting the risk.

There is no evidence here of criminal conduct, fraud, sabotage, or intentional service degradation. There is no basis to claim that PagerDuty concealed accepted-event loss; its account expressly says previously accepted events and data were not lost, while separately reporting rejection and delay. Those statements should be preserved together. A critical analysis becomes weaker, not stronger, when it converts carefully scoped operational facts into unsupported allegations.

The source also does not independently verify the completion of long-term remediation. The rollback is a confirmed incident action. Cluster stabilization is confirmed in the company's account. The status process used a backup manual route. But a postmortem's proposed or implied learning is not the same as a later audit showing controls operating over time. Reviewers should distinguish repair intention, implementation evidence, and effectiveness evidence. Only the incident-day interventions are closed by the published record.

Customer-side evidence remains another unknown. A customer with local request logs, PagerDuty response codes, webhook identifiers, and notification timestamps could reconstruct its own exposure more precisely. That evidence is outside the published postmortem. The absence of a public customer-by-customer ledger should not be misread as proof that no one was harmed, nor should it be treated as permission to invent harm. It is a reason to keep conclusions at the level the evidence supports.

Recovery Is Complete Only When the Cause, the Queue, and the Record Are Stable

PagerDuty's August 28 outages show why service restoration is only one layer of recovery. At 10:10 UTC, services were normal, but the causal trigger had not been identified and removed. During restoration, stale messages and duplicate outputs could still reach customers. Public status communication had already lagged internal response. The system was functioning, yet the cause, the queue, and the external record were not all equally settled.

By the second incident, responders had a known stabilization playbook. They mitigated impact faster, found the abnormal traffic source, and rolled back the offending code. Full restoration at 20:24 closed the operational day in a stronger state than the first restoration had. That does not erase the earlier failure. It clarifies it: capacity and cluster recovery can buy time, but causal recovery requires removing the workload that made the capacity inadequate.

The case also shows why an incident-management provider cannot define reliability only as eventual delivery. Its product sits inside customers' response clocks. A notification that is preserved but late, a webhook that is duplicated, an event that is rejected, and a status update that remains unpublished impose different risks. Each needs its own evidence and recovery behavior. Aggregating them into one uptime number would make the accountability chain harder to see.

PagerDuty's publication of detailed metrics and a concrete causal mechanism is part of responsible disclosure. It makes scrutiny possible. The remaining test is whether the organization can demonstrate that its controls now observe the behavior that mattered: producer creation rate, broker memory and garbage collection in relation to changes, backlog semantics, independent status publication, and causal closure before an incident is declared fully recovered. Those are not demands for perfection. They are demands for evidence aligned with the controls the company actually holds.

The central lesson is restrained but demanding. A platform used to manage incidents became a source of uncertainty during two of its own. The documented root cause was a logical error in PagerDuty feature code. Shared Kafka architecture and incomplete cross-layer detection contributed. The first response restored service without removing the trigger. The second linked abnormal traffic to the feature and rolled it back. No unsupported allegation is needed.

The chronology itself shows where accountability belongs: with the parties that could see, constrain, communicate, and remove the risk—and most of those decisive controls were PagerDuty's.

Sources