How a failure is confirmed

One failed run never wakes you. Before we tell you anything, we make sure the failure is real: we try again, then we ask the Monitor’s other locations to look.

The second try

When a run fails, the location that made it tries once more, 5 to 10 seconds later. The wait is a little different each time, so the locations watching the same address do not all come back to it in the same second.

When the second try gets through, nothing happens. A blip is not an outage, and you hear nothing about it.

Asking the other locations

When the second try fails too, we ask the Monitor’s other locations to look, at once. Each of them tries once, and we wait up to 30 seconds for their answers. A location that has not answered by then is left out.

An Incident opens when most of the locations that answered agree that it failed, the first one included. An Incident never opens on fewer than 2 answers, so one location alone can never open one on a Monitor that has more.

The Incident card says how many agreed, as “2 of 3 locations agree”: the locations that saw it fail, out of every location that was asked, the first one included. Once the Incident has ended, it says “agreed”.

When the others reach it and only the first location saw it fail, no Incident opens and you hear nothing. If that one location keeps failing while the others get through, it becomes a Regional outage.

Seeing how it was decided

Open the Monitor’s Events tab and expand an Incident. How uup decided lists each step with its time: what the first location saw, its second try, which locations were asked and what each of them saw, and what we decided.

Recovery is confirmed the same way

An Incident ends the way it opened. When a location gets through again, the other locations are asked to look too, and the Incident ends once most of those that answered agree that it is up. A Monitor is never called up again because one location got through.

When too few locations can answer

Sometimes too few of our locations can answer: fewer than 2 answers come in. Then we decide nothing and send nothing. No Incident opens and no alert goes out, and How uup decided says “No answer to go on”. When it happens while an Incident is open, the Incident stays open until the locations can answer again.

That hour can show as not measured in your uptime, because the fault is ours. A not-measured hour counts neither for nor against you.

One location

A Monitor that probes from one location has nobody to ask. Free probes from 1 location, so it tries again before it tells you: when the second try fails too, the Incident opens and you are told. Its Incident card says “One location”.

A Heartbeat Monitor has no locations at all. Your job calls us, so its Incident opens when no call comes in time, with nobody to ask.

Next

Incidents says what an Incident is, what you see while one is open, and the ways it ends.