When many things fail at once

A failure seldom comes alone. When a server stops answering, its certificate can no longer be read either. Several of your Monitors can fail in the same minute. And a site that keeps going down and coming back would send an alert each time. In each case you get fewer messages, each one saying more.

The cause inside one message

A Monitor has an uptime Check, and it can have a certificate, domain and performance Check too. When the uptime Check fails, the others can fail with it: a certificate cannot be read from a server that does not answer. That second failure is not news.

So while the uptime Check has an open Incident, an Incident on another Check of the same Monitor sends nothing of its own. The uptime alert is the one you hear, and what its failed request saw is said inside it: “acme.com is down: its certificate expired”. Where a request stopped says how to read it.

The other Check’s Incident is still recorded. On the Events tab it is folded under the outage it belongs to, marked “held: explained by the outage above”.

When the uptime Incident ends, each held Check runs again at once:

• If it passes, nothing about it is ever sent.
• If it still fails, its own alert is sent, after the recovery message.

An Incident on another Check that opens just before the uptime Check fails waits for the next uptime run before it alerts. If the uptime Check fails meanwhile, it is held in the same way.

Many Monitors at once

When several Monitors in one Project fail together, you hear about them once.

The first Incident in the Project alerts at once, on its own, as usual. Every Incident in the same Project that opens within 90 seconds of it waits. Then:

• If no other Incident opened, nothing more happens.
• If one other opened, its alert is sent as usual. Two failures together are two pieces of news.
• If 3 Monitors or more failed together, counting the first, each Channel gets one message about them all: “acme.com, api.acme.com and shop.acme.com went down together”, with a line for each saying what we saw. It ends: “You will get one more message when the last of them recovers.”

From then on they travel together. A Repeat alert lists only the ones still down, so the list grows shorter as they recover. One recovery message comes when the last of them is back.

Each Channel’s message lists only the Monitors whose rules send down alerts to it. Nobody hears about a Monitor that was never pointed at them.

They are grouped by Project and by the clock, never by a cause. They failed at the same time; the message does not say why, or that they share anything. Grouping changes only how you hear about it. Each Monitor’s downtime, its uptime percentage and its Status page are exactly as they would be alone.

A Check that is snoozed never joins, and nor does a Check held behind its own Monitor’s uptime Incident.

On the dashboard, the Project’s page shows one line while they are open: “5 Monitors went down together at 14:02 UTC”, under it “They were sent to you in one message.”, and Acknowledge, which acknowledges them all. Each Monitor’s Incident on its Events tab says “Sent in one message: 5 Monitors went down together at 14:02 UTC”.

Flapping

A Check that fails, recovers and fails again soon after is flapping. When a Check’s Incident opens within 20 minutes of its previous Incident opening, you get one message in place of the usual alert: “acme.com keeps failing and recovering”. It says: “You will get no more messages while it keeps changing, and one more message once it has stayed up.”

That Incident stays open while the Check goes on failing and recovering. It ends once every run, from every location, has passed for 15 minutes, and then one recovery message is sent. The Incident is marked flapping on the Events tab and in the list of Monitors.

The same rule runs for a Regional outage, at the one location it is about.

Only failures count. An Incident you ended yourself, by pausing the Monitor, turning the Check off or resolving it by hand, is not counted, and nor is one we reported in error. Pause, snooze, acknowledge, disable says what each of those does.