Article
Diagnosing Pending Pods with Polestar KCM
Detailed Article · From Root Cause Identification to In-Product Remediation
An operator's job continues even after an application is deployed. When traffic surges, processing capacity must be increased, and when processes crash, they must be restarted. When deploying a new version, operators must also ensure that existing services remain uninterrupted.
As the number of services and servers grows, managing these tasks manually becomes increasingly difficult. Kubernetes automates operations by placing containerized applications across multiple servers and maintaining their declared state.
Even in an automated environment, questions remain. Why does a container keep restarting? Why won't a new pod run even though CPU utilization is low? For stable operations, you must be able to understand the processes and outcomes of automation.
In this article, we outline key monitoring elements to watch in Kubernetes and walk through the workflow in Polestar KCM—from identifying the cause of a Pending pod to performing necessary remediations and additional diagnostics directly within the product.
Operational Tasks Handled by Kubernetes
Containers bundle applications together with their execution environments, making them easy to deploy. As the number of containers grows, you must manage where to run them, how many to maintain, and how to recover them when they fail.
In Kubernetes, the basic unit for running containers is the Pod. The servers on which pods run are Nodes, and nodes combined with control plane components form a Cluster. Workload resources such as Deployments declare the desired number of pods and deployment configurations.
For example, if you configure a deployment to maintain three pods, Kubernetes reconciles the actual state to match that declaration. If a container fails, it restarts it according to policy; if a managed pod fails, it creates a replacement pod. Root causes requiring separate remediation, such as application bugs, must be investigated by operators. Official Kubernetes Documentation on Self-Healing

Figure 1. Monitoring bridges Kubernetes automation and operator judgment.
What to Verify Through Monitoring
It is difficult to assess the health of a service based on execution state alone. Even if a container restarts, memory leaks can cause recurring issues, and a running pod might not yet be ready to receive traffic. This is why status, performance metrics, and events must be examined together.
Observation Target | Key Checkpoints | Questions Operators Must Answer |
Clusters and Nodes | Node status, CPU/Memory, requests vs. allocatable capacity | Is there sufficient capacity to schedule new pods? |
Workloads and Pods | Ready status, replica count, restarts, termination reasons | Are the required pods running and ready to handle requests? |
Services and Networking | Connection topologies and traffic flow | Which section should communication troubleshooting start from? |
Storage | Volume utilization and PVC binding status | Is storage blocking execution or processing? |
Events and Logs | Error messages, timestamps, and recurrence patterns | What happened before and after the issue occurred? |
Configuration and History | Configuration changes and pod replacement times | How does the current state differ from the state during the incident? |
This table summarizes key perspectives to check during Kubernetes operations. The availability of individual metric items varies depending on the product version, deployment environment, and collection settings. The case study below focuses on pod status, scheduling events, CPU requests, and available node capacity.
Why a Pod Won't Run Even with Low CPU Utilization
Suppose a newly deployed pod is stuck in a Pending state. The node's CPU utilization is only 20%, which makes it seem like there is plenty of room. However, when the scheduler evaluates CPU resource suitability, its decision is based on resource requests, not current utilization.
The compute resources a node can allocate to pods are called allocatable resources. You can determine whether a node can accommodate a new pod only by taking into account the requests of already scheduled pods. Even if actual usage is low, high existing requests can prevent additional scheduling. Official Kubernetes Documentation on Resource Management for Pods and Containers
In the following example, each node has an allocatable CPU capacity of 4 cores, and the sum of requests from existing pods is 3 cores. Summing the request-based available capacity across the three nodes yields 3 cores, but there is no single node capable of accommodating a new pod requesting 2 cores. (Assuming memory and other scheduling conditions are met.)

Figure 2. Total cluster capacity differs from the capacity available on a single node. Numbers are examples for illustrative purposes.
The core takeaway of this case is distinguishing between actual usage, total cluster headroom, and per-node headroom. In actual scheduling, memory, node selectors, taints and tolerations, and volume constraints also come into play, so it is essential to check the raw event messages as well.
The Workflow from Root Cause Identification to Remediation with Polestar KCM
The screens below are examples constructed by injecting mock data into Polestar KCM's actual Pod Map renderer, Events tab, Schedule Capacity panel, and Command Execution drawer. Step navigation buttons, raw text highlight panels, and instructional callouts were added for this article. Targets, progress states, and command outputs are mock data; no commands or modifications were executed on an actual cluster.
Scenario Item | Example Value |
Cluster / Namespace | demo-commerce / commerce |
Target Pod | checkout-api-7f9b6-demo |
Total Pods | 13 — Running: 12, Pending: 1 |
Nodes | worker-a, worker-b, worker-c — all Ready |
Per-Node CPU | Allocatable: 4 cores, Existing requests: 3 cores, Actual utilization: 20% |
New Pod Requests | CPU: 2 cores, Memory: 1 GiB |
Observation Window | October 26, 2026, 10:00–10:30 KST |
Failure Events | 6 FailedScheduling events between 10:10–10:15 |
1 Identify the Investigation Target from Pod Status
First, locate the Pending pod on the Pod Map. In this example, 12 running pods and 1 waiting pod are clearly distinguished. Identifying the selected pod's name and namespace clarifies what target to query for subsequent events.

Figure 3. Locating the orange Pending pod. The selected pod is unscheduled and requests 2 CPU cores.
What needs to be verified here is the fact that the target pod has not yet been scheduled to a node. Because Pending can also appear during initialization stages other than scheduling wait, do not jump to the conclusion of CPU shortage based on status alone. Check the pod's events in the next step.
2 Check the Cause of Scheduling Failure in Events
Narrowing down the events to the target pod reveals recurring FailedScheduling occurrences. The example message reads as follows:
0/3 nodes are available: 3 Insufficient cpu.

Figure 4. Checking the FailedScheduling event and the resource shortage classification. The raw message is displayed in the bottom panel for easier reading.
This message indicates that none of the three nodes evaluated by the scheduler met the CPU requirements. It does not mean actual CPU utilization is at 100%. You must compare the resource cited in the raw message with the requests of the new pod.
Additionally, an event count of 6 on the screen does not mean there are six pods waiting. In this scenario, six scheduling failure events occurred for a single pod. Distinguishing between target count and event occurrence count prevents misinterpreting the scope of an incident.
3 Compare Requests with Per-Node Scheduling Capacity
Next, check the scheduling capacity. In our example, the total allocatable CPU is 12 cores, and the sum of requests from currently scheduled pods is 9 cores. While total cluster headroom is 3 cores, the maximum headroom on any single node is only 1 core.

Figure 5. Viewing both the total headroom of 3 cores and the maximum single-node headroom of 1 core. No node can accommodate the new pod's 2-core request.
Comparison Item | Value | Interpretation |
Total Allocatable CPU | 12 cores | CPU allocatable to pods across three nodes |
Total Existing Pod Requests | 9 cores | Sum of CPU requests of currently scheduled pods |
Total Request-Based Headroom | 3 cores | Aggregated cluster headroom |
Max Single-Node Headroom | 1 core | Upper limit of CPU requests any single node can accept in this scenario |
New Pod Request | 2 cores | Exceeds single-node headroom |
This comparison pinpoints the exact problem: although actual CPU utilization on each node is low, there is no single node with enough capacity to accept the new pod's request. The N−1 indicator on the screen provides supplementary context regarding total capacity risk if one node fails. Note that CPU and memory headroom alone does not guarantee all scheduling constraints are met.
4 Execute Remediations In-Product and Verify Results
Once the cause is identified, it is time to decide on remediation. You can add nodes, adjust workload placement, or review resource settings if requests are excessive relative to actual load. In environments where an agent is connected, Polestar KCM provides commands tailored to user permissions and target conditions, enabling a direct transition from investigation to remediation.
If resource tuning is required, select `Adjust CPU/Memory` from the command execution button in Pod Details. In this example, we assume that reviewing peak load and application profiles determined a CPU request of 1 core to be optimal. Change the checkout container request in the checkout-api Deployment from 2 cores to 1 core, while keeping the CPU limit at 2 cores and preserving memory settings. Note that lowering requests simply because current CPU utilization is low is not recommended.
After verifying the modified values and target, applying the changes updates the workload's pod template. Changes are not applied to just a single selected pod; rather, they trigger pod replacements according to the workload's deployment strategy. The screen allows you to track the progress of the control task and the Ready status of the new generation of pods. Official Kubernetes Documentation on Deployment Mechanics

Figure 6. Example showing the actual CPU/Memory adjustment screen with request updated from 2 to 1 core, along with the new generation waiting for Ready status. The 1/3 progress is mock data and does not indicate recovery completion.
When you need to inspect container internals more deeply, you can use the `Execute Command` feature. By selecting a container in a running pod and entering a command, you can execute it non-interactively via the agent and review stdout, stderr, and exit codes. Below is an example of running ps -ef in the gateway container of gateway-01 (Running state) to inspect running processes. This is distinct from executing commands on a Pending pod that has not yet been scheduled to a node.

Figure 7. Selecting a container, entering a command, and reviewing execution output on the actual command execution screen. The ps -ef output is mock data; in practice, the container must have the corresponding binary installed.
In this way, targets identified during monitoring can be investigated further within the product, and supported control operations can be executed seamlessly. Operators save the trouble of re-entering targets and commands into disparate tools, accelerating incident response.
After remediation, verify whether the new pod has been scheduled to a node, satisfies Running status and Ready conditions, and whether any new FailedScheduling events appear. Finally, check application response times and error rates. An operational response is complete only when both command execution results and service normalization have been verified. This article uses mock data to illustrate the remediation interface and verification workflow, and does not represent actual recovery results.
Examine History and Ingestion Status Alongside Current State
In Kubernetes, pods are frequently replaced during deployment and self-healing cycles. If the problematic pod has already been deleted by the time of investigation, it is difficult to reconstruct the incident solely from currently running resources.
Pod history in Polestar KCM can be used to investigate archived records of deleted pods, their historical specifications, replacement timestamps, and performance metrics. Reviewing events and configuration history together helps reconstruct context before and after a failure.
It is also critical to verify whether telemetry data is actively being ingested. A lack of new events could mean that no events actually occurred, or that metric collection has stopped. Polestar KCM separates data collection states—such as ingestion lag or disconnection—from resource health, allowing each to be assessed independently.
From Detection to Root Cause, Remediation, and Verification
Kubernetes automates repetitive operational tasks. Operators must understand problems that arise during that automation and determine the appropriate actions to take.
In this case study, we located the target using the Pod Map, identified the reason for scheduling failure through events, and compared per-node scheduling headroom against pod requests. We then walked through executing necessary changes via the in-product resource adjustment screen, monitoring progress, and running commands in an active container for deeper diagnostics.
By connecting status, performance, events, and history with agent-based command execution, Polestar KCM supports operational workflows seamlessly—from identifying root causes to executing remediations and verifying outcomes.
Author: JH Son | Research Team 2, Nkia R&D Center




