Article

Diagnosing Pending Pods with Polestar KCM

Kubernetes Pending Pod Diagnosis Article Featured Image
Kubernetes Pending Pod Diagnosis Article Featured Image


Detailed Article · From Root Cause Identification to In-Product Remediation


An operator's job continues even after an application is deployed. When traffic surges, processing capacity must be increased, and when processes crash, they must be restarted. When deploying a new version, operators must also ensure that existing services remain uninterrupted.

As the number of services and servers grows, managing these tasks manually becomes increasingly difficult. Kubernetes automates operations by placing containerized applications across multiple servers and maintaining their declared state.

Even in an automated environment, questions remain. Why does a container keep restarting? Why won't a new pod run even though CPU utilization is low? For stable operations, you must be able to understand the processes and outcomes of automation.

In this article, we outline key monitoring elements to watch in Kubernetes and walk through the workflow in Polestar KCM—from identifying the cause of a Pending pod to performing necessary remediations and additional diagnostics directly within the product.




Operational Tasks Handled by Kubernetes


Containers bundle applications together with their execution environments, making them easy to deploy. As the number of containers grows, you must manage where to run them, how many to maintain, and how to recover them when they fail.

In Kubernetes, the basic unit for running containers is the Pod. The servers on which pods run are Nodes, and nodes combined with control plane components form a Cluster. Workload resources such as Deployments declare the desired number of pods and deployment configurations.

For example, if you configure a deployment to maintain three pods, Kubernetes reconciles the actual state to match that declaration. If a container fails, it restarts it according to policy; if a managed pod fails, it creates a replacement pod. Root causes requiring separate remediation, such as application bugs, must be investigated by operators. Official Kubernetes Documentation on Self-Healing

쿠버네티스가 맡아 주는 운영 작업 → 쿠버네티스 운영 흐름 – 원하는 상태 선언, 쿠버네티스 자동화, Polestar KCM으로 관측

Figure 1. Monitoring bridges Kubernetes automation and operator judgment.




What to Verify Through Monitoring


It is difficult to assess the health of a service based on execution state alone. Even if a container restarts, memory leaks can cause recurring issues, and a running pod might not yet be ready to receive traffic. This is why status, performance metrics, and events must be examined together.

Observation Target

Key Checkpoints

Questions Operators Must Answer

Clusters and Nodes

Node status, CPU/Memory, requests vs. allocatable capacity

Is there sufficient capacity to schedule new pods?

Workloads and Pods

Ready status, replica count, restarts, termination reasons

Are the required pods running and ready to handle requests?

Services and Networking

Connection topologies and traffic flow

Which section should communication troubleshooting start from?

Storage

Volume utilization and PVC binding status

Is storage blocking execution or processing?

Events and Logs

Error messages, timestamps, and recurrence patterns

What happened before and after the issue occurred?

Configuration and History

Configuration changes and pod replacement times

How does the current state differ from the state during the incident?

This table summarizes key perspectives to check during Kubernetes operations. The availability of individual metric items varies depending on the product version, deployment environment, and collection settings. The case study below focuses on pod status, scheduling events, CPU requests, and available node capacity.




Why a Pod Won't Run Even with Low CPU Utilization


Suppose a newly deployed pod is stuck in a Pending state. The node's CPU utilization is only 20%, which makes it seem like there is plenty of room. However, when the scheduler evaluates CPU resource suitability, its decision is based on resource requests, not current utilization.

The compute resources a node can allocate to pods are called allocatable resources. You can determine whether a node can accommodate a new pod only by taking into account the requests of already scheduled pods. Even if actual usage is low, high existing requests can prevent additional scheduling. Official Kubernetes Documentation on Resource Management for Pods and Containers

In the following example, each node has an allocatable CPU capacity of 4 cores, and the sum of requests from existing pods is 3 cores. Summing the request-based available capacity across the three nodes yields 3 cores, but there is no single node capable of accommodating a new pod requesting 2 cores. (Assuming memory and other scheduling conditions are met.)


CPU 사용률은 낮은데 → 노드별 CPU 여유 예시 – 총여유 3코어지만 2코어 요청 파드를 받을 노드가 없어 Pending

Figure 2. Total cluster capacity differs from the capacity available on a single node. Numbers are examples for illustrative purposes.

The core takeaway of this case is distinguishing between actual usage, total cluster headroom, and per-node headroom. In actual scheduling, memory, node selectors, taints and tolerations, and volume constraints also come into play, so it is essential to check the raw event messages as well.




The Workflow from Root Cause Identification to Remediation with Polestar KCM


The screens below are examples constructed by injecting mock data into Polestar KCM's actual Pod Map renderer, Events tab, Schedule Capacity panel, and Command Execution drawer. Step navigation buttons, raw text highlight panels, and instructional callouts were added for this article. Targets, progress states, and command outputs are mock data; no commands or modifications were executed on an actual cluster.

Scenario Item

Example Value

Cluster / Namespace

demo-commerce / commerce

Target Pod

checkout-api-7f9b6-demo

Total Pods

13 — Running: 12, Pending: 1

Nodes

worker-a, worker-b, worker-c — all Ready

Per-Node CPU

Allocatable: 4 cores, Existing requests: 3 cores, Actual utilization: 20%

New Pod Requests

CPU: 2 cores, Memory: 1 GiB

Observation Window

October 26, 2026, 10:00–10:30 KST

Failure Events

6 FailedScheduling events between 10:10–10:15




1 Identify the Investigation Target from Pod Status


First, locate the Pending pod on the Pod Map. In this example, 12 running pods and 1 waiting pod are clearly distinguished. Identifying the selected pod's name and namespace clarifies what target to query for subsequent events.


파드 상태에서 → Polestar KCM 파드 상태 화면 – 전체 13개 파드 중 Pending 1개 확인

Figure 3. Locating the orange Pending pod. The selected pod is unscheduled and requests 2 CPU cores.

What needs to be verified here is the fact that the target pod has not yet been scheduled to a node. Because Pending can also appear during initialization stages other than scheduling wait, do not jump to the conclusion of CPU shortage based on status alone. Check the pod's events in the next step.




2 Check the Cause of Scheduling Failure in Events

Narrowing down the events to the target pod reveals recurring FailedScheduling occurrences. The example message reads as follows:

0/3 nodes are available: 3 Insufficient cpu.


이벤트에서 → Polestar KCM 이벤트 화면 – FailedScheduling, ‘0/3 nodes are available: 3 Insufficient cpu’

Figure 4. Checking the FailedScheduling event and the resource shortage classification. The raw message is displayed in the bottom panel for easier reading.

This message indicates that none of the three nodes evaluated by the scheduler met the CPU requirements. It does not mean actual CPU utilization is at 100%. You must compare the resource cited in the raw message with the requests of the new pod.

Additionally, an event count of 6 on the screen does not mean there are six pods waiting. In this scenario, six scheduling failure events occurred for a single pod. Distinguishing between target count and event occurrence count prevents misinterpreting the scope of an incident.




3 Compare Requests with Per-Node Scheduling Capacity


Next, check the scheduling capacity. In our example, the total allocatable CPU is 12 cores, and the sum of requests from currently scheduled pods is 9 cores. While total cluster headroom is 3 cores, the maximum headroom on any single node is only 1 core.


요청량과 노드별 → Polestar KCM 스케줄 여유 비교 – 노드별 CPU requests와 할당 가능량 비교

Figure 5. Viewing both the total headroom of 3 cores and the maximum single-node headroom of 1 core. No node can accommodate the new pod's 2-core request.


Comparison Item

Value

Interpretation

Total Allocatable CPU

12 cores

CPU allocatable to pods across three nodes

Total Existing Pod Requests

9 cores

Sum of CPU requests of currently scheduled pods

Total Request-Based Headroom

3 cores

Aggregated cluster headroom

Max Single-Node Headroom

1 core

Upper limit of CPU requests any single node can accept in this scenario

New Pod Request

2 cores

Exceeds single-node headroom

This comparison pinpoints the exact problem: although actual CPU utilization on each node is low, there is no single node with enough capacity to accept the new pod's request. The N−1 indicator on the screen provides supplementary context regarding total capacity risk if one node fails. Note that CPU and memory headroom alone does not guarantee all scheduling constraints are met.




4 Execute Remediations In-Product and Verify Results


Once the cause is identified, it is time to decide on remediation. You can add nodes, adjust workload placement, or review resource settings if requests are excessive relative to actual load. In environments where an agent is connected, Polestar KCM provides commands tailored to user permissions and target conditions, enabling a direct transition from investigation to remediation.

If resource tuning is required, select `Adjust CPU/Memory` from the command execution button in Pod Details. In this example, we assume that reviewing peak load and application profiles determined a CPU request of 1 core to be optimal. Change the checkout container request in the checkout-api Deployment from 2 cores to 1 core, while keeping the CPU limit at 2 cores and preserving memory settings. Note that lowering requests simply because current CPU utilization is low is not recommended.

After verifying the modified values and target, applying the changes updates the workload's pod template. Changes are not applied to just a single selected pod; rather, they trigger pod replacements according to the workload's deployment strategy. The screen allows you to track the progress of the control task and the Ready status of the new generation of pods. Official Kubernetes Documentation on Deployment Mechanics


제품 안에서 (첫 번째) → Polestar KCM 명령 실행 화면 – 파드 CPU·메모리 request/limit 조정

Figure 6. Example showing the actual CPU/Memory adjustment screen with request updated from 2 to 1 core, along with the new generation waiting for Ready status. The 1/3 progress is mock data and does not indicate recovery completion.


When you need to inspect container internals more deeply, you can use the `Execute Command` feature. By selecting a container in a running pod and entering a command, you can execute it non-interactively via the agent and review stdout, stderr, and exit codes. Below is an example of running ps -ef in the gateway container of gateway-01 (Running state) to inspect running processes. This is distinct from executing commands on a Pending pod that has not yet been scheduled to a node.


제품 안에서 (두 번째) → Polestar KCM 명령 실행 화면 – 실행 중인 파드에서 ps -ef 결과 확인

Figure 7. Selecting a container, entering a command, and reviewing execution output on the actual command execution screen. The ps -ef output is mock data; in practice, the container must have the corresponding binary installed.


In this way, targets identified during monitoring can be investigated further within the product, and supported control operations can be executed seamlessly. Operators save the trouble of re-entering targets and commands into disparate tools, accelerating incident response.

After remediation, verify whether the new pod has been scheduled to a node, satisfies Running status and Ready conditions, and whether any new FailedScheduling events appear. Finally, check application response times and error rates. An operational response is complete only when both command execution results and service normalization have been verified. This article uses mock data to illustrate the remediation interface and verification workflow, and does not represent actual recovery results.




Examine History and Ingestion Status Alongside Current State


In Kubernetes, pods are frequently replaced during deployment and self-healing cycles. If the problematic pod has already been deleted by the time of investigation, it is difficult to reconstruct the incident solely from currently running resources.

Pod history in Polestar KCM can be used to investigate archived records of deleted pods, their historical specifications, replacement timestamps, and performance metrics. Reviewing events and configuration history together helps reconstruct context before and after a failure.

It is also critical to verify whether telemetry data is actively being ingested. A lack of new events could mean that no events actually occurred, or that metric collection has stopped. Polestar KCM separates data collection states—such as ingestion lag or disconnection—from resource health, allowing each to be assessed independently.




From Detection to Root Cause, Remediation, and Verification


Kubernetes automates repetitive operational tasks. Operators must understand problems that arise during that automation and determine the appropriate actions to take.

In this case study, we located the target using the Pod Map, identified the reason for scheduling failure through events, and compared per-node scheduling headroom against pod requests. We then walked through executing necessary changes via the in-product resource adjustment screen, monitoring progress, and running commands in an active container for deeper diagnostics.

By connecting status, performance, events, and history with agent-based command execution, Polestar KCM supports operational workflows seamlessly—from identifying root causes to executing remediations and verifying outcomes.




Author: JH Son | Research Team 2, Nkia R&D Center