4digital.cloud4digital.cloud
← Engineering

Early problem detection

Monitoring with independent oversight

In short: Monitoring built as code, with rules checked by tests and three complementary layers of oversight.

01 / For your organisation

A failure noticed first by customers makes it harder for the team to respond promptly. Good monitoring lets you tell the absence of detected problems apart from the loss of the ability to detect them.

Separate oversight of monitoring helps detect a loss of visibility within configured intervals and thresholds. The alert rules the team relies on are checked by tests.

02 / Delivered in 4digital’s environment

In 4digital’s own environment I built the monitoring from the ground up, with requirements set by measuring what actually needs to be observed. The work rested on four principles:

  • Alerts checked like code — every rule has tests showing that the alert fires and clears, and tests against deliberately broken rules confirm that the tests themselves catch errors.
  • The observer outside the observed — a separate witness checks the monitoring, and a signal from outside checks the whole environment.
  • No data means unknown — when there is no fresh data about a host, its state is unknown, not healthy.
  • Progress over presence — I judge a data-processing service by whether data keeps arriving, not by whether the process exists.

The monitoring is code deployed by automation from the repository. Access uses a separate identity for each role, and rules follow from the inventory, so adding a host needs no change in logic. I went through every rule of the previous stack with a decision: keep, generalise, replace or retire.

03 / Outcome

The monitoring has three complementary layers of oversight: a witness that confirms the whole alerting chain every minute, a signed heartbeat from every host received outside the environment, and an external monitor that checks the public site by the content of the response, not just the status code. A separate guard detects a silent stop in data processing and recovers it automatically.

Value: it is clear which alerts actually work and who watches the monitoring itself, and no alerts can be told apart from a loss of detection.

04 / Selected evidence

  • Rule acceptance tests caught all 16 deliberately introduced rule changes.
  • A single witness answer confirms the entry point, rule evaluation and alert delivery at once.
  • A service outside the environment notifies when one host goes quiet, when all go quiet and when it loses sight of the signals itself.
  • A data-processing guard with tests that include negative controls.
Source of experience
4digital’s own environment
Discipline
Observability
Technology areas
Prometheus · Alertmanager · Blackbox exporter · Rule tests (promtool) · Ansible · nginx with TLS · systemd · Cloudflare Workers