Skip to main content
Legia

IT Infrastructure & Cloud

Server Monitoring: Metrics to Watch and Their Alert Thresholds

Good server monitoring is not the one with the most charts, but the one whose alerts always mean something. This guide sorts the core metrics, starting thresholds from common tools, and how to tune them.

SEO and Content Specialist

8 min read

Blue network cables leading to a switch panel

Short answer: useful server monitoring watches five things: service availability from the user’s point of view, CPU and load, memory, disk space and inodes, and the clock and backup status. Start from a tool’s default thresholds, for example disk at 80% and 90% in the Zabbix template, or CPU and memory at 90% for 15 minutes in Prometheus rules, then tune them with your own server’s data. Send urgent alerts only for conditions that need action right away, and leave the rest to tickets or a dashboard.

Many companies already run a monitoring tool but still learn about a server problem from user complaints. The cause is usually one of two things: the metrics that matter are not monitored, or the alerts are so noisy that people ignore them. This guide covers the core metrics for server monitoring, example starting thresholds from the documentation of common tools, and how to sort which alerts should wake someone up. Sizing the server being monitored is covered in Calculating Internal Application Server Specifications.

Server monitoring starts from what users feel

Monitoring should answer two questions: what is broken, and why. Google’s Site Reliability Engineering book calls the first the symptom and the second the cause, and recommends putting more effort into catching symptoms than causes. The Prometheus alerting guide agrees: for systems that serve users, alert on latency and error rate as high in the service layer as possible.

The same book describes four golden signals:

SignalMeaningExample on an office server
LatencyTime to serve one requestTime for an internal application to show a page
TrafficHow much demand is placed on the systemRequests per second or active users
ErrorsRate of failing requestsServer error codes in the application
SaturationHow “full” the service isDisk, memory, and CPU nearing their limits

Saturation matters because, according to that source, many systems slow down well before resource use reaches 100%. Most of the server metrics below are saturation and cause signals. So pair them with one symptom check: a probe that requests a page or service port from outside the server, as the blackbox exporter does over HTTP, TCP, ICMP, or DNS.

Illustration: server

Core metrics and example starting thresholds

The table below summarizes basic Linux server metrics with the default thresholds of two common tools: the “Linux by Zabbix agent” template and the alert rules of the node-mixin from the Prometheus node_exporter project. These numbers are starting points from the tool authors, not a standard that applies to every server.

MetricZabbix (template default)Prometheus node-mixin
CPU usageAbove 90%, 5-minute windowAbove 90% for 15 minutes, info level
Load average per coreAbove 1.5Above 2 per core for 15 minutes
Memory usageAbove 90%; available memory below 20 MBAbove 90% for 15 minutes
Free swapBelow 50%No similar rule in this list
Disk spaceUsed above 80% (warning), above 90% (critical)Free below 40% and predicted to run out within 24 hours (warning); free below 20% and predicted to run out within 4 hours (critical); free below 5% and 3% without prediction
Free inodesBelow 20% (warning), below 10% (critical)Below 40% and 20% with prediction; below 5% and 3% without prediction
Server clock offsetNot part of this template summaryOffset above 0.05 seconds for 10 minutes

A few notes on reading this table:

  1. A threshold only means something together with its duration. The Prometheus rules use a 15-minute delay for CPU and memory, so a brief spike does not trigger an alert. The Prometheus guide says the same: allow slack for small blips.
  2. Disk has two kinds of thresholds. A percentage threshold catches a disk that is already nearly full. A predictive threshold, which estimates when the disk will run out from its fill rate, gives more time to act. Servers whose logs or databases grow quickly suit the second kind better.
  3. Inodes are monitored too. A disk can refuse new files while space remains if inodes run out, for example in a directory holding millions of small files.
  4. A small clock offset has real effects. A drifting clock makes tracing logs across servers harder and can disrupt time-based authentication.

Windows servers have equivalent metrics. The concepts are the same, but counter names and default thresholds follow the tool you use.

Three alert levels: wake someone, open a ticket, or just look

Google’s SRE book stresses that every alert that wakes someone must be actionable. An alert that only calls for a robotic response should not be an urgent alert. Email alerts, according to the book, have limited value and tend to become noise, so problems that are not yet critical are better shown on a dashboard. Prometheus adds that you should avoid alerts that give you nothing to do.

From those principles, these three levels are practical:

LevelWhen to useExample
Urgent (send to the person on duty)The service has stopped or will stop within hoursService probe failing repeatedly, critical disk, server sending no data
Working-hours ticketAction needed, but not necessarily tonightDisk past the warning threshold, memory use trending up, certificate nearing expiry
Dashboard onlyInformation for analysis, not actionBrief CPU spike, daily traffic, usage trends

Mapping metrics to these levels is a decision for you and your provider, not a number from a tool. One rule of thumb: if the recipient of an alert does not know what to do, the alert needs to be rewritten or downgraded.

How to tune thresholds with your own data

Default thresholds suit the first week. After that, tune them in this order:

  1. Record normal conditions for at least a few weeks, including monthly peaks such as book closing or payroll.
  2. Install the default thresholds and run them in log-only mode, without sending urgent alerts.
  3. Review the alerts that appear. For each one, ask whether anyone took action. If not, raise the threshold, lengthen the delay, or lower the level.
  4. Add a delay to metrics that fluctuate, so brief spikes do not trigger alerts.
  5. Review every quarter and after major changes, such as a new application or a server migration.

Steps 1 and 5 are recommended practice here, not requirements from the tools’ documentation. The point is the same as in the Prometheus guide: a few alerts that always mean something are more useful than many alerts that are often ignored.

What server monitoring often misses

  • Backups. Monitor that backup jobs succeed and the output exists, not just that the scheduler is running. A backup that succeeds in the log may still not be restorable, so test restores as in Restore Testing for Server Backups and apply a 3-2-1 Backup Strategy.
  • Certificate and domain expiry. Expiry is invisible to CPU or memory metrics, but it makes a service unusable. Raise tickets well ahead of the date.
  • Monitoring from outside the network. A monitor running on the same server dies with the server. Prometheus recommends testing the whole alert path, and monitoring the monitoring itself.
  • Admin access logs. Repeated failed logins or logins at unusual hours are security signals, not performance signals. Securing access is covered in Securing Admin Access to Servers.
  • An owner. Every urgent alert needs one name or one on-call rota. An alert sent to “everyone” is usually handled by no one.

Server monitoring checklist for an office

  1. List all servers and running services, with their owners.
  2. Set up a probe from outside the server for every important service.
  3. Monitor CPU, load, memory, swap, disk, inodes, and clock on every server.
  4. Monitor the status and results of backup jobs.
  5. Monitor certificate and domain expiry.
  6. Define three alert levels and who receives each one.
  7. Start from the tool’s default thresholds, then tune from at least a few weeks of data.
  8. Review alerts that nobody acted on every quarter.

How Legia does it

In the Server Installation & Management service, installing monitoring is a stage of its own: Legia enables monitoring, early warnings, and automatic backups, then tests them. This stage takes 3 to 5 working days. The tools listed on the service page include Prometheus, Grafana, and Zabbix. Monitoring runs automatically around the clock and sends an alert as soon as there is a disruption, while handling by the team follows the service hours and severity levels written in your contract. In monthly management, the report covers server availability, incidents, security updates installed, and resource usage. The other services in this pillar are on the IT Infrastructure & Cloud page.

Frequently asked questions

What is the minimum set of metrics to monitor?

Start with service availability, CPU or load, memory, disk space, and backup status. Those five catch most of the reasons a server stops serving. Add inodes, the server clock, and certificates once the basics are stable.

Is an 80% disk threshold always right?

No. It is the default warning threshold of the Zabbix template, and the Prometheus rules use a prediction of when the disk will run out instead. A disk at 80% on a server that barely grows means something different from 80% on a server writing logs every second. Tune it from the fill rate of your own server.

Why isn’t a CPU alert sent as soon as CPU is high?

Because brief spikes are normal, for example during backups or updates. The default rules add a delay, such as 15 minutes in the Prometheus node-mixin, so only conditions that persist are reported.

Is monitoring from inside the server enough?

Not yet. A monitor running on the same server reports nothing when that server is down. Add a check from outside that reaches the service like a user would, and monitor the monitor itself too.

Sources

Category IT Infrastructure & Cloud

Keep reading

Related articles

All Articles