Summary

  • Inngest’s October 9 network failure and October 10 database-pool failure were separate events, but both interrupted execution through dependencies shared across its service.
  • The hardest customer-facing failure was not only delayed work: some internally failed runs stayed Running, so failure handlers and waiting parent functions were not notified. Customers had to cancel or replay them.

A durable workflow is usually sold through a simple promise: work can wait, resume after interruption and retain completed steps. Inngest’s account of October 9–11 shows the clauses hidden inside that promise. Where is run state stored? Which queues share a processing path? What status is written when an internal dependency fails? Who learns that a run will not resume by itself?

Inngest described two incidents, not one continuing outage. At 21:05 UTC on October 9, a provider lost connectivity between one data centre and Inngest’s other centres, as well as outbound Internet access. Inngest said the provider’s preliminary explanation was an error during a DDoS-protection configuration change and that it was still waiting for the full report. Connectivity returned at 21:49. On October 10, the company’s two PgBouncer pools lost all connections to the primary database at 14:00 and again at 15:30. Queue processing and run scheduling stalled for most accounts.

A mitigation restored processing, then introduced intermittent delays that continued until cron scheduling was moved back at 04:17 UTC on October 11; gateway routing was improved the following day. The timeline matters because each event had a different failure mechanism and recovery path.

The October 9 architecture made a local boundary failure travel. Inngest said it ran across three data centres: accounts could be processed in one place while their run state lived in another; account configuration and the primary database sat behind still another provider. A failed cross-centre path meant affected workers could not fetch account details or save state. But the effect did not stop with those accounts. Repeatedly failing work built a backlog in a shared event-processing pipeline, delaying execution for accounts that had no direct dependency on the isolated centre. Accounts on dedicated event pipelines were not affected.

This is the distinction between a component’s blast radius and the queue’s blast radius: shared scheduling can widen a failure after the original network dependency is no longer relevant to each delayed account.

The next day exposed a different coupling. Inngest traced the pool collapse to short lookup deadlines interacting with client cancellation and PgBouncer. Some queries were still waiting for a pool connection when their callers timed out. The driver closed the client connection; later, PgBouncer ran the abandoned query and closed the database connection rather than returning it to the pool. Because replacement connections opened one at a time, destruction outpaced replenishment.

The collapses coincided with top- and bottom-of-hour cron traffic, which roughly tripled query volume, and a recently deployed set of lookups that began failing at scale. Restarting the poolers did not reliably fix the cycle; a larger pool limit did not address how slowly new connections could be opened.

The workaround also carried a new dependency. Moving services to poolers beside the database restored processing between 16:10 and 16:25 UTC. That route then traversed network-address-translation gateways that silently dropped some connections. A query on a dropped connection could wait up to 16 minutes before failing, with cron functions especially visible to users. The incident record says Inngest moved cron scheduling back to the original poolers at 04:17 UTC on October 11 and improved gateway routing on October 12.

A mitigation is itself a topology change: it can remove one bottleneck while adding a different, less visible timeout boundary.

The state machine left a more consequential question. Inngest said internal failures consumed configured retries. When a step failed permanently because of an internal error, some runs remained Running rather than moving to Failed. That meant an onFailure handler did not run, and a parent waiting through step.invoke did not receive a failure. The company said those runs would not continue on their own; customers had to cancel or replay them. A healthy worker pool therefore did not automatically mean every interrupted business process had reached a terminal state.

This differs from ordinary workflow resumption. Inngest documents that completed step results can be reused when a function resumes, while steps are retried independently. But its incident note says customer replay starts the run again from the beginning. Previously completed steps can then run again. That does not prove a payment, message or database write was duplicated. It does put the burden on the operator to inspect the trace and confirm that repeated effects are safe. Inngest’s own idempotency guidance warns that an external service can accept an action before the workflow records its result; a retry can then make the same call again.

Durable checkpoints preserve progress inside the orchestration system. They do not by themselves make every external effect exactly once.

Inngest said it had shipped several corrections: lookups no longer cancel queries still waiting for a pool connection; an internal permanent failure is marked failed; and if run state cannot be read at that moment, work stays queued and resumes when state becomes available. It also reported improved gateway routing and faster status-page updates. The larger architectural response—consolidating onto a single data centre—was still under way, with reliability work and published service SLOs planned through year-end. The report does not describe the finished topology or its independent disaster-recovery design.

Sources