Article
Server hardware monitoring, covering failures the OS can't see
Failures begin where the OS agent cannot see
The agent is quiet, yet the server halts
When an issue occurs on a physical server, the monitoring screen suddenly floods with "Agent Disconnected" alarms from the VMs running on top of it. Looking at the alarms alone, it easily resembles a network outage, taking time to pinpoint the root cause.
Tracing the cause often leads to hardware. One of the redundant power supply units (PSUs) fails first, and when the remaining one fails as well, the entire server goes down. However, the first failure is usually never logged by any OS.
What we commonly call "server monitoring" mostly consists of metrics observed by an agent running inside the OS. CPU, memory, disk, and processes are clearly visible, but the ground the OS stands on remains invisible.
Hardware monitoring is about looking at that ground. In this article, we examine where the hardware signals unseen by the OS come from, why those signals protect a scope far broader than a single server, and the architectural decisions we made sequentially while embedding this into our monitoring product.
TL;DR of this article ① The OS agent and BMC are two distinct eyes that do not replace one another. ② A hardware failure propagates to all VMs and OSs above it, and the earliest signals originate outside the OS. ③ Rather than reading many sensors, it is crucial to view and alert in units of replaceable components.
Another computer inside the server, the BMC
Most data center server motherboards are equipped with a small management computer called a Baseboard Management Controller (BMC). Different manufacturers call it by different names: Dell calls it iDRAC, HPE calls it iLO, and Lenovo calls it XClarity Controller.
The core characteristic of a BMC is that it is isolated from the OS.
•It runs on its own chip and firmware. It operates even without the host CPU.
•It stays alive via standby power. Even if the server's power button is turned off, it responds as long as the power cable is plugged in.
•It communicates via a dedicated management network. Even if the service network goes down or the OS halts, the management port remains open separately.
•It is directly attached to motherboard sensors. It reads fan RPMs, PSU status, temperature and voltage across various sections, and logs hardware events.
Thus, there are two ways to monitor a server: the In-band path sent by an agent inside the OS over the service network, and the Out-of-band path answered by the BMC over the management network.

Figure 1. In-band and Out-of-band — Two ways to observe a server
The OS agent responds through the service network while the OS is running; the BMC responds through the management network regardless of the OS.
What is seen inside the OS vs. what is visible only outside the OS
Both paths observe the same server, but at different layers. The OS agent answers "How is this OS using resources?" while the BMC answers "Is this machine physically sound?"
Category | OS Agent (In-band) | BMC (Out-of-band) |
Scope of Observation | CPU, memory, disk utilization, processes, logs | Fan rotation, PSU status, temperature/voltage by section, power status |
Operating Condition | OS must be booted and the agent must be running | Requires only the power cable to be connected |
When the Server Halts | Collection stops. Only the last recorded values remain | Continues reading power/component status and System Event Logs (SEL) |
Questions It Answers Well | "Why did it slow down?" "What is consuming resources?" | "Why did it shut off?" "What needs to be replaced?" |
Blind Spots | Causes originating from hardware | Inside the OS and applications |
The last row of the table is crucial. The two are partners that cover each other's blind spots; one cannot replace the other. If you infer hardware status using only the agent, failure arrives merely as the end result of "connection lost."
When one piece of hardware stops, everything above it stops
With virtualization and containers becoming standard, the burden carried by a single physical server has grown significantly. A hypervisor sits on top of one machine, multiple VMs run on top of that, and OSs and containers run inside them.
In this architecture, hardware failure propagates upward. If all PSUs fail, or if fans fail and power is cut off for thermal protection, every VM running on top halts simultaneously.
At this point, the view from the OS side looks like this:
•Agents on each VM leave a "Connection Lost" status. Alarms pour in equal to the number of VMs.
•None of the alarms contain the information that "the root cause is the PSU of the same physical server."
•Operators must trace backward from the symptoms to find the cause.
Conversely, the BMC sees the single root cause. And usually, that cause is visible long before the service stops.

Figure 2. Blast Radius — A PSU failure on 1 physical server halts 4 VMs
Hardware monitoring offers two values. The first is the earliest signal. The failure of a single PSU appears while the service is still running fine, and replacing it then prevents the second failure.
The other is the blast radius. Knowing which physical server has an issue allows you to preemptively migrate VMs running on it or group simultaneously triggered alarms into a single root cause.
Failures do not happen out of nowhere
Most server hardware features redundancy and protection mechanisms. Therefore, failures do not happen all at once, but rather progress through several stages. There are two primary systems:
•Cooling System: If one fan degrades or stops, the remaining fans spin faster to compensate. If temperatures still rise, the CPU throttles itself, and eventually powers down for protection.
•Power System: Even if one of the two PSUs stops, the server continues to run. However, from the moment redundancy is broken, the server relies entirely on the remaining PSU.
The problem is that most of this progression passes by as if nothing happened as far as the OS is concerned. What the OS experiences is merely slowdown due to throttling, followed by the final "connection lost."

Figure 3. Progression of Failure — 5 Stages of Cooling and Power Systems and Detection Entities
Hardware monitoring targets the three stages on the left. In this interval, the service is intact, and resolving it requires replacing only a single fan or PSU.
What to look for by component and how to read it
While the BMC reports many values, operationally meaningful questions converge into one for each component. Determining what it means before looking at the value itself makes defining thresholds natural as well.
Component | Representative Signal | Operational Question | Points to Note When Reading |
Temperature | Sensors by section: CPU, intake, exhaust, etc. | Where is the hottest spot? | Averages are meaningless. If intake temperature rises alongside, it is an HVAC issue, not the server |
Fan | Rotation speed (RPM) or % of maximum speed | Is any fan stopped? | 0 means stopped. A sudden increase in other fans' speeds is also a sign of a neighboring fan failure |
PSU | Status (OK/Warning/Critical), output power, rated capacity | Is redundancy intact? | The server runs even if one stops. Hence, it is missed most often |
Voltage | Voltage by rail | Has it deviated from the nominal rating? | Nominal ratings differ by rail, so they cannot be evaluated with a single threshold |
System Power | Total power consumption (W) | Is it unusual compared to normal? | Trends and ratio against rated capacity are more meaningful than absolute values |
Power/Overall Status | Power on/off, overall assessment determined by BMC | Is this machine healthy right now? | Fastest and most reliable as it is evaluated by the manufacturer's internal criteria |
Event Log (SEL) | Hardware event records | What happened? | Only newly generated entries should be viewed. Reading historical records every time causes alarm floods |
A notable point is that the units of values vary by device. Some devices report fan metrics in RPM, while others report in %. This difference is addressed again in the next section.
IPMI and Redfish: Different answers to the same question
There are largely two ways to communicate with a BMC: IPMI, which has been used for a long time, and Redfish, its successor.
Item | IPMI 2.0 | Redfish |
Communication | UDP 623, RMCP+ sessions | HTTPS, REST API |
Data Structure | Reads sensor data records (SDR) one by one | JSON documents structured by component |
Strengths | Widely supported, even on older equipment | Rich and human-readable information, such as inventory and ratings |
Weaknesses | Value names and classifications differ by vendor | Legacy equipment either lacks support or supports it only partially |
Having a standard does not mean the answers are identical. Here is the difference when we read servers from two different vendors using the same Redfish protocol.
Item | Dell PowerEdge R730 (iDRAC) | HPE ProLiant DL360 Gen10 (iLO5) |
Number of Temp Sensors | 4 | 54 |
Fan Value Unit | RPM | % of maximum speed |
PSU Naming | Includes numbers, like "PS1 Status" | All PSUs have identical names; numbers are separate properties |
IPMI is even more fragmented. Even for the exact same PSU, the status sensor comes as "PS1 Status" while the power sensor arrives as "PS1 Input Power" under separate names. If you display these raw values on screen, a single PSU appears as if it were two.
Ultimately, the challenge of hardware monitoring lies not in "reading values," but in normalizing disparate responses into a single model.
Focus on components, not sensors
If you turn all 54 temperature sensors of an HPE server into 54 monitoring targets, you end up with 50,000 temperature targets for 1,000 servers. Conversely, if you attach all values to a single server entity, neither the tree view nor alarms can specify "which fan has stopped."
We established a single principle: Count by the units that field engineers actually replace (FRU, Field Replaceable Unit). What is pulled and replaced from a server is a fan or a PSU, not an individual temperature sensor.
Component | Monitoring Target | Reason |
Fan | Each individual fan | It is a replacement unit and few in number. Must alert by name, e.g., "Fan 2 Stopped" |
PSU | Each individual PSU | It is a replacement unit, and knowing which one stopped is key to evaluating redundancy |
Temperature | One target per server; individual sensor values inside | Sensors are not replacement units. Alarms are determined by the hottest sensor |
Voltage | One target per server | Same reason as temperature |
Power/Power State/Overall Status | The server itself | These are values for the machine as a whole |
Once defined this way, the units of tree nodes, performance charts, and alarms aligned seamlessly. Clicking "Fan 2" in the tree displays the chart for that fan, and alarms trigger under the name of "Fan 2."
Unified naming across different protocols
The second decision was metric naming. Previously, IPMI and Redfish produced different names for the same temperature metric, requiring two separate sets of alarm policies. Now, both paths produce metrics under identical names.
Names follow the <domain>.<component>.<metric> convention. In names like ipmi.fan.speed, ipmi.psu.status, and ipmi.temperature.celsius, the middle segment indicates the component type. The collector decides which component to attach the value to solely based on the name.
Disparate answers from different vendors are then normalized to match these names.
•If fan data arrives in RPM, it is categorized as fan.speed; if in %, as fan.percent. Values with different units are never mixed under one name.
•For PSUs, numbers are extracted from the name to normalize to PSU1. "PS1 Status" and "PS1 Input Power" map to the same PSU.
•Empty slots (Absent) are not created as PSUs, as this would result in permanently empty nodes.

Figure 4. Observation Unit Model — Vendor Value → Normalization → Component-level Target
Thanks to this model, a single alarm policy is all that is needed. IPMI and Redfish servers can be grouped together to deploy the same policy.
Collection successful, 0 alarms
Most design revisions were triggered by symptoms of "no errors, but empty data." In hardware monitoring, "collection successful" does not mean "actively monitored."
Symptom | Cause | What Was Changed |
Collection succeeds, but fan charts are empty only for HPE devices | Fan values not in RPM were discarded. HPE reports in % | Split metrics by unit and emit all of them |
A single PSU appears twice on the screen | Sensor names for the same PSU differed | Extract numbers from names to consolidate them |
Alarm template deployed, but 0 rules created | If the deployment step failed to link metrics to collectors, it silently skipped the item | Provide default collection policies together to establish the link |
"Alarm if Status is Critical" becomes an empty policy | Threshold parsing only recognized numbers, discarding string conditions | Create status metrics separately with string comparison conditions |
The common thread among all four was that success signals masked failures. Consequently, our verification criteria shifted from "Are there no errors?" to "How many alarm rules were actually generated after deployment?"
Out-of-the-box default alarms
Based on this experience, we included a "Server BMC Hardware Default Alarms" template in the product. Thresholds are generated automatically from the metric catalog.
Target | Alarm Condition | Evaluation Criteria |
Temperature | 70 / 80 / 90 ℃ (Notice / Warning / Critical) | Hottest sensor |
Fan | Stopped (0) | Each individual fan |
PSU | Status Warning / Critical | Each individual PSU |
Overall Server Status | Warning / Critical | BMC assessment |
Power | Off | Server |
Voltage and power were excluded from the default alarms. They are meaningful only when evaluated as "% of rated capacity," but many devices do not provide rated capacity values. Rather than providing unfounded numbers as defaults, we left them for operators to configure according to their hardware.
Remaining challenges
•Alerting on collection failure itself: Notifying an inability to connect to the BMC via a separate availability alarm. Distinguishing communication failures from hardware failures is necessary.
•Alarms based on System Event Logs (SEL): Currently, we aggregate counts of new events; archiving raw text and alerting by event type are the next steps.
•Connecting physical servers to their hosted VMs: Linking which BMC alarm affects which VMs within virtualization monitoring. The goal is to present the blast radius shown in Figure 2 directly on the screen.
How it looks in Polestar 10 Nova
The model described above is reflected directly in the monitoring interface. Expanding a BMC device in the server tree displays fans, PSUs, temperature, and voltage as child nodes, and the performance tab uses the same chart and table views as other monitoring domains.
[Screen 1] Hardware Status — Overall health and summary view covering temperature, power, cooling, etc.

[Screen 2] PSU Alarm — An anomaly alarm triggered on a single PSU

Conclusion — Looking at the ground
The OS agent looks inside the server. However, everything inside stands on top of a single fan and a single PSU on the outside. If the ground shakes, every OS running above it shakes along with it.
The starting point for properly viewing that ground was not reading numerous sensors. It required defining first what an operator counts as a single unit, what they replace, and what they want to be alerted on.
In previous articles, we segmented a single GPU card into multiple slices, and grouped scattered facility equipment into a single topology for FMS. In server hardware, we regrouped dozens of sensors into replaceable component units. Although the approaches vary, the question remains the same: "What does the recipient of this alarm do next?"
References
•DMTF, Redfish Specification — Redfish Data Model and REST API
•Intel et al., Intelligent Platform Management Interface (IPMI) Specification v2.0 — RMCP+, SDR, SEL
•BMC manuals by respective vendors — Dell iDRAC, HPE iLO, Lenovo XClarity Controller
Author: DW Jang | Nkia Research Lab




