Availability monitoring without silent failures

When successive hosts in a group stopped working, the chart showed degradation, but alerting did not reflect it correctly.

The initial hypothesis concerned notification grouping. Before changing the alerting, it had to be checked what the metric behind the alert actually calculated.

I rebuilt the monitoring around four principles:

• Verifying what the metric means — checking which question the metric actually answers before changing the notification logic.
• Availability calculated per group — aggregation based on the actual metadata, so that the result describes the availability of individual functional groups rather than the whole.
• Zero as a result — series with a zero value stay in the calculation, so a complete group outage produces 0% rather than an empty result.
• Explicit detection limits — cases the metric cannot detect, such as a complete absence of series after a host is removed from service discovery, are documented explicitly.

The analysis showed that an incorrect operation on metric labels combined hosts into one group: the system calculated global availability rather than the availability of individual functional groups. The result looked credible while answering a different question.

In addition, the calculation removed unavailable series from the numerator, so during a complete outage of a group whose hosts still reported a zero status, the result could disappear instead of becoming 0% — the alert could be lost in the most critical scenario.

I rebuilt the aggregation on the actual metadata and calculated availability as the mean of binary values. I separated the alerting logic, data presentation and individual-host diagnostics. Degradation of individual groups is detected correctly, and a complete group outage with status series still present produces 0% rather than silence.

Value: the alert rests on a metric that answers the right question — the availability of a specific group — even in the most critical scenario.