Blog

Why One Failed Check Should Never Wake You Up

Alert fatigue kills on-call teams. How failure thresholds, recovery detection and maintenance windows cut false positives without hiding real outages.

Upmonora teamSeptember 8, 20262 min read

Overview - Why One Failed Check Should Never Wake You Up

The internet is noisy. Packets drop, DNS resolvers hiccup and a server can pause for garbage collection. If every one of those blips pages someone at 3 a.m., people learn to ignore alerts - and then they miss the real one.

Consecutive-failure thresholds

The single most effective control is simple: only open an incident after N consecutive failures. Upmonora defaults to two. With a one-minute interval that means a genuine outage is confirmed within about two minutes, while a single transient error is recorded but never alerted.

Choosing the right threshold

  • Critical checkout or API: threshold 2 with a 30–60 second interval.
  • Marketing site or blog: threshold 2–3 with a 5-minute interval.
  • Known flaky upstream: threshold 3–4, and investigate the flakiness separately.

Recovery should be instant

Thresholds apply to going down, not coming back. One successful check resolves the incident and sends a recovery alert, so your downtime numbers stay accurate.

Use maintenance windows

Planned deploys are not outages. Schedule a maintenance window and Upmonora keeps checking, but opens no incidents and sends no downtime alerts while it is active.

Separate "we failed" from "you failed"

A monitoring worker crashing is not your website going down. Upmonora retries its own failures with back-off and never marks a site as down because of an internal error - a distinction many teams only discover after a false 3 a.m. page.