Article

Server hardware monitoring, covering failures the OS can't see

Server Hardware Monitoring Illustration – A server monitoring the motherboard BMC chip and Power Supply Unit (PSU) warning lights
Server Hardware Monitoring Illustration – A server monitoring the motherboard BMC chip and Power Supply Unit (PSU) warning lights

Failures begin where the OS agent cannot see



The agent is quiet, yet the server halts


When an issue occurs on a physical server, the monitoring screen suddenly floods with "Agent Disconnected" alarms from the VMs running on top of it. Looking at the alarms alone, it easily resembles a network outage, taking time to pinpoint the root cause.

Tracing the cause often leads to hardware. One of the redundant power supply units (PSUs) fails first, and when the remaining one fails as well, the entire server goes down. However, the first failure is usually never logged by any OS.

What we commonly call "server monitoring" mostly consists of metrics observed by an agent running inside the OS. CPU, memory, disk, and processes are clearly visible, but the ground the OS stands on remains invisible.

Hardware monitoring is about looking at that ground. In this article, we examine where the hardware signals unseen by the OS come from, why those signals protect a scope far broader than a single server, and the architectural decisions we made sequentially while embedding this into our monitoring product.


TL;DR of this article ① The OS agent and BMC are two distinct eyes that do not replace one another. ② A hardware failure propagates to all VMs and OSs above it, and the earliest signals originate outside the OS. ③ Rather than reading many sensors, it is crucial to view and alert in units of replaceable components.




Another computer inside the server, the BMC


Most data center server motherboards are equipped with a small management computer called a Baseboard Management Controller (BMC). Different manufacturers call it by different names: Dell calls it iDRAC, HPE calls it iLO, and Lenovo calls it XClarity Controller.

The core characteristic of a BMC is that it is isolated from the OS.

•It runs on its own chip and firmware. It operates even without the host CPU.

•It stays alive via standby power. Even if the server's power button is turned off, it responds as long as the power cable is plugged in.

•It communicates via a dedicated management network. Even if the service network goes down or the OS halts, the management port remains open separately.

•It is directly attached to motherboard sensors. It reads fan RPMs, PSU status, temperature and voltage across various sections, and logs hardware events.

Thus, there are two ways to monitor a server: the In-band path sent by an agent inside the OS over the service network, and the Out-of-band path answered by the BMC over the management network.


서버 안의 또 하나의 컴퓨터, BMC → 서버 구조도 – OS 에이전트는 In-band 서비스망으로, BMC(iDRAC·iLO·XClarity)는 Out-of-band 관리망(IPMI·Redfish)으로 관제 서버에 데이터 전송


Figure 1. In-band and Out-of-band — Two ways to observe a server

The OS agent responds through the service network while the OS is running; the BMC responds through the management network regardless of the OS.




What is seen inside the OS vs. what is visible only outside the OS


Both paths observe the same server, but at different layers. The OS agent answers "How is this OS using resources?" while the BMC answers "Is this machine physically sound?"


Category

OS Agent (In-band)

BMC (Out-of-band)

Scope of Observation

CPU, memory, disk utilization, processes, logs

Fan rotation, PSU status, temperature/voltage by section, power status

Operating Condition

OS must be booted and the agent must be running

Requires only the power cable to be connected

When the Server Halts

Collection stops. Only the last recorded values remain

Continues reading power/component status and System Event Logs (SEL)

Questions It Answers Well

"Why did it slow down?" "What is consuming resources?"

"Why did it shut off?" "What needs to be replaced?"

Blind Spots

Causes originating from hardware

Inside the OS and applications


The last row of the table is crucial. The two are partners that cover each other's blind spots; one cannot replace the other. If you infer hardware status using only the agent, failure arrives merely as the end result of "connection lost."




When one piece of hardware stops, everything above it stops


With virtualization and containers becoming standard, the burden carried by a single physical server has grown significantly. A hypervisor sits on top of one machine, multiple VMs run on top of that, and OSs and containers run inside them.

In this architecture, hardware failure propagates upward. If all PSUs fail, or if fans fail and power is cut off for thermal protection, every VM running on top halts simultaneously.

At this point, the view from the OS side looks like this:

•Agents on each VM leave a "Connection Lost" status. Alarms pour in equal to the number of VMs.

•None of the alarms contain the information that "the root cause is the PSU of the same physical server."

•Operators must trace backward from the symptoms to find the cause.

Conversely, the BMC sees the single root cause. And usually, that cause is visible long before the service stops.


하드웨어 하나가 멈추면 → 물리 서버의 PSU 고장이 하이퍼바이저 위 VM 4대로 번지는 구조 – OS에는 ‘연결 끊김’만, BMC에는 원인이 보인다


Figure 2. Blast Radius — A PSU failure on 1 physical server halts 4 VMs

Hardware monitoring offers two values. The first is the earliest signal. The failure of a single PSU appears while the service is still running fine, and replacing it then prevents the second failure.

The other is the blast radius. Knowing which physical server has an issue allows you to preemptively migrate VMs running on it or group simultaneously triggered alarms into a single root cause.




Failures do not happen out of nowhere


Most server hardware features redundancy and protection mechanisms. Therefore, failures do not happen all at once, but rather progress through several stages. There are two primary systems:

•Cooling System: If one fan degrades or stops, the remaining fans spin faster to compensate. If temperatures still rise, the CPU throttles itself, and eventually powers down for protection.

•Power System: Even if one of the two PSUs stops, the server continues to run. However, from the moment redundancy is broken, the server relies entirely on the remaining PSU.

The problem is that most of this progression passes by as if nothing happened as far as the OS is concerned. What the OS experiences is merely slowdown due to throttling, followed by the final "connection lost."


고장은 갑자기 오지 않는다 → 냉각·전원 계통 고장 진행 단계 – BMC는 팬·PSU 이상을 OS보다 먼저 감지


Figure 3. Progression of Failure — 5 Stages of Cooling and Power Systems and Detection Entities

Hardware monitoring targets the three stages on the left. In this interval, the service is intact, and resolving it requires replacing only a single fan or PSU.




What to look for by component and how to read it


While the BMC reports many values, operationally meaningful questions converge into one for each component. Determining what it means before looking at the value itself makes defining thresholds natural as well.


Component

Representative Signal

Operational Question

Points to Note When Reading

Temperature

Sensors by section: CPU, intake, exhaust, etc.

Where is the hottest spot?

Averages are meaningless. If intake temperature rises alongside, it is an HVAC issue, not the server

Fan

Rotation speed (RPM) or % of maximum speed

Is any fan stopped?

0 means stopped. A sudden increase in other fans' speeds is also a sign of a neighboring fan failure

PSU

Status (OK/Warning/Critical), output power, rated capacity

Is redundancy intact?

The server runs even if one stops. Hence, it is missed most often

Voltage

Voltage by rail

Has it deviated from the nominal rating?

Nominal ratings differ by rail, so they cannot be evaluated with a single threshold

System Power

Total power consumption (W)

Is it unusual compared to normal?

Trends and ratio against rated capacity are more meaningful than absolute values

Power/Overall Status

Power on/off, overall assessment determined by BMC

Is this machine healthy right now?

Fastest and most reliable as it is evaluated by the manufacturer's internal criteria

Event Log (SEL)

Hardware event records

What happened?

Only newly generated entries should be viewed. Reading historical records every time causes alarm floods


A notable point is that the units of values vary by device. Some devices report fan metrics in RPM, while others report in %. This difference is addressed again in the next section.




IPMI and Redfish: Different answers to the same question


There are largely two ways to communicate with a BMC: IPMI, which has been used for a long time, and Redfish, its successor.


Item

IPMI 2.0

Redfish

Communication

UDP 623, RMCP+ sessions

HTTPS, REST API

Data Structure

Reads sensor data records (SDR) one by one

JSON documents structured by component

Strengths

Widely supported, even on older equipment

Rich and human-readable information, such as inventory and ratings

Weaknesses

Value names and classifications differ by vendor

Legacy equipment either lacks support or supports it only partially


Having a standard does not mean the answers are identical. Here is the difference when we read servers from two different vendors using the same Redfish protocol.



Item

Dell PowerEdge R730 (iDRAC)

HPE ProLiant DL360 Gen10 (iLO5)

Number of Temp Sensors

4

54

Fan Value Unit

RPM

% of maximum speed

PSU Naming

Includes numbers, like "PS1 Status"

All PSUs have identical names; numbers are separate properties


IPMI is even more fragmented. Even for the exact same PSU, the status sensor comes as "PS1 Status" while the power sensor arrives as "PS1 Input Power" under separate names. If you display these raw values on screen, a single PSU appears as if it were two.

Ultimately, the challenge of hardware monitoring lies not in "reading values," but in normalizing disparate responses into a single model.




Focus on components, not sensors


If you turn all 54 temperature sensors of an HPE server into 54 monitoring targets, you end up with 50,000 temperature targets for 1,000 servers. Conversely, if you attach all values to a single server entity, neither the tree view nor alarms can specify "which fan has stopped."

We established a single principle: Count by the units that field engineers actually replace (FRU, Field Replaceable Unit). What is pulled and replaced from a server is a fan or a PSU, not an individual temperature sensor.


Component

Monitoring Target

Reason

Fan

Each individual fan

It is a replacement unit and few in number. Must alert by name, e.g., "Fan 2 Stopped"

PSU

Each individual PSU

It is a replacement unit, and knowing which one stopped is key to evaluating redundancy

Temperature

One target per server; individual sensor values inside

Sensors are not replacement units. Alarms are determined by the hottest sensor

Voltage

One target per server

Same reason as temperature

Power/Power State/Overall Status

The server itself

These are values for the machine as a whole


Once defined this way, the units of tree nodes, performance charts, and alarms aligned seamlessly. Clicking "Fan 2" in the tree displays the chart for that fan, and alarms trigger under the name of "Fan 2."




Unified naming across different protocols


The second decision was metric naming. Previously, IPMI and Redfish produced different names for the same temperature metric, requiring two separate sets of alarm policies. Now, both paths produce metrics under identical names.

Names follow the <domain>.<component>.<metric> convention. In names like ipmi.fan.speed, ipmi.psu.status, and ipmi.temperature.celsius, the middle segment indicates the component type. The collector decides which component to attach the value to solely based on the name.

Disparate answers from different vendors are then normalized to match these names.

•If fan data arrives in RPM, it is categorized as fan.speed; if in %, as fan.percent. Values with different units are never mixed under one name.

•For PSUs, numbers are extracted from the name to normalize to PSU1. "PS1 Status" and "PS1 Input Power" map to the same PSU.

•Empty slots (Absent) are not created as PSUs, as this would result in permanently empty nodes.


프로토콜이 달라도 이름은 하나 → Dell iDRAC·HPE iLO5·IPMI 센서 값을 팬·PSU·온도 단위 모델로 정규화하는 과정


Figure 4. Observation Unit Model — Vendor Value → Normalization → Component-level Target

Thanks to this model, a single alarm policy is all that is needed. IPMI and Redfish servers can be grouped together to deploy the same policy.




Collection successful, 0 alarms


Most design revisions were triggered by symptoms of "no errors, but empty data." In hardware monitoring, "collection successful" does not mean "actively monitored."


Symptom

Cause

What Was Changed

Collection succeeds, but fan charts are empty only for HPE devices

Fan values not in RPM were discarded. HPE reports in %

Split metrics by unit and emit all of them

A single PSU appears twice on the screen

Sensor names for the same PSU differed

Extract numbers from names to consolidate them

Alarm template deployed, but 0 rules created

If the deployment step failed to link metrics to collectors, it silently skipped the item

Provide default collection policies together to establish the link

"Alarm if Status is Critical" becomes an empty policy

Threshold parsing only recognized numbers, discarding string conditions

Create status metrics separately with string comparison conditions


The common thread among all four was that success signals masked failures. Consequently, our verification criteria shifted from "Are there no errors?" to "How many alarm rules were actually generated after deployment?"





Out-of-the-box default alarms


Based on this experience, we included a "Server BMC Hardware Default Alarms" template in the product. Thresholds are generated automatically from the metric catalog.


Target

Alarm Condition

Evaluation Criteria

Temperature

70 / 80 / 90 ℃ (Notice / Warning / Critical)

Hottest sensor

Fan

Stopped (0)

Each individual fan

PSU

Status Warning / Critical

Each individual PSU

Overall Server Status

Warning / Critical

BMC assessment

Power

Off

Server


Voltage and power were excluded from the default alarms. They are meaningful only when evaluated as "% of rated capacity," but many devices do not provide rated capacity values. Rather than providing unfounded numbers as defaults, we left them for operators to configure according to their hardware.




Remaining challenges

•Alerting on collection failure itself: Notifying an inability to connect to the BMC via a separate availability alarm. Distinguishing communication failures from hardware failures is necessary.

•Alarms based on System Event Logs (SEL): Currently, we aggregate counts of new events; archiving raw text and alerting by event type are the next steps.

•Connecting physical servers to their hosted VMs: Linking which BMC alarm affects which VMs within virtualization monitoring. The goal is to present the blast radius shown in Figure 2 directly on the screen.





How it looks in Polestar 10 Nova


The model described above is reflected directly in the monitoring interface. Expanding a BMC device in the server tree displays fans, PSUs, temperature, and voltage as child nodes, and the performance tab uses the same chart and table views as other monitoring domains.


[Screen 1] Hardware Status — Overall health and summary view covering temperature, power, cooling, etc.

Polestar 10 Nova에서는 (첫 번째) → Polestar 10 Nova 서버 요약 화면 – 시스템·CPU 온도, 전력 소비, 팬·PSU 상태와 헬스 Critical 표시


[Screen 2] PSU Alarm — An anomaly alarm triggered on a single PSU

Polestar 10 Nova에서는 (두 번째) → Polestar 10 Nova 전력 성능 화면 – PSU 1/2 정상, PS2 장애 표시




Conclusion — Looking at the ground


The OS agent looks inside the server. However, everything inside stands on top of a single fan and a single PSU on the outside. If the ground shakes, every OS running above it shakes along with it.

The starting point for properly viewing that ground was not reading numerous sensors. It required defining first what an operator counts as a single unit, what they replace, and what they want to be alerted on.

In previous articles, we segmented a single GPU card into multiple slices, and grouped scattered facility equipment into a single topology for FMS. In server hardware, we regrouped dozens of sensors into replaceable component units. Although the approaches vary, the question remains the same: "What does the recipient of this alarm do next?"






References

•DMTF, Redfish Specification — Redfish Data Model and REST API

•Intel et al., Intelligent Platform Management Interface (IPMI) Specification v2.0 — RMCP+, SDR, SEL

•BMC manuals by respective vendors — Dell iDRAC, HPE iLO, Lenovo XClarity Controller







Author: DW Jang | Nkia Research Lab