Summary

  • Infrastructure can stay healthy while a configuration change makes an automated service produce the wrong result.
  • Shruthi Sepuri’s answer is to test the workflow’s output in production, compare it with expected behavior, and feed the evidence back into release testing.

A green dashboard is a confident witness with a narrow field of view. It can testify that servers answered, latency stayed low and error rates remained quiet. It cannot testify that the answer a service gave was the right one.

That gap is the subject of Shruthi Sepuri’s NANOG 98 presentation, “From Testing to Reliability: Detecting Silent Failures in Production Systems.” NANOG’s programme scheduled the 30-minute talk in Miami on 19 October 2026. The official abstract points to a familiar trap in complex systems: configuration, rules and dependencies can change what an automated workflow decides without disturbing the infrastructure metrics operators already watch.

Sepuri’s slides make the trap visible as a scenario. The infrastructure side stays green; a rule engine fails, a decision comes out wrong, a validation step is skipped and the audit trail has a gap. The public material names no affected service or real incident, so the example is a model of failure rather than a case history. Its point is still sharp: “the service is up” and “the service did the right thing” are different claims.

The proposed repair is to measure the path between input and consequence. Sepuri lists output correctness, completeness of configuration coverage, shifts in output distributions, and whether rules execute as intended against known cases. She also recommends replaying synthetic service traffic through live logic, comparing post-change outputs with a known-good baseline, and feeding production behavior back into the test set.

This is more than adding another graph to an already crowded wall. A useful check names the decision the service is supposed to make, the inputs that can change it, and the evidence that would reveal a wrong answer. For a rule-driven workflow, that might mean validating required fields before execution, checking the rule path, and recording the result with enough context to trace a configuration change.

A distribution anomaly is a lead, not a verdict. Traffic, policy or demand may have changed for a good reason. The alert needs a change record, an owner who can explain the expected outcome, and a comparison that separates planned behavior from drift. The metric becomes operational only when someone can act on it.

Sepuri’s argument widens the meaning of reliability: monitor the system that is running, not only the machines that host it. The test suite should reach production through safe replay and return with evidence. Otherwise, an immaculate dashboard can certify the wrong decision all the way downstream.

Sources: NANOG 98 session page · official presentation slides · official speaker portrait.