Uninterruptible Sleep State (D State) in Kubernetes

Context

During a routine workday, alerts for our IBM Product Master deployment began firing across Microsoft Teams and Outlook. The application was an older deployment running on an OpenShift cluster that had originated on OpenShift 4.8.x and had been upgraded across multiple minor releases to 4.18.x.

Around 15 projects shared this cluster, but only IBM Product Master showed service-level outages and failed monitoring checks.

Investigating

Checking the OpenShift events in the affected namespace showed repeated warnings from kubelet attempting to stop the pods without success. Despite the failures, the pods were still reported in the Running state. Visiting the exposed OpenShift routes produced blank white screens. The application pod logs showed no active errors, primarily because requests were hanging before the application layer could process or log them.

The application team confirmed that their WebSphere Liberty logs showed no active exceptions or crash traces. The deployment was an older manual configuration without an official IBM operator to manage the application lifecycle.

The primary clue was kubelet's inability to stop or evict the pods. Because container runtimes cannot complete a stop command until processes leave uninterruptible system calls, I accessed the underlying node directly:

1oc debug node/<node-name> -- chroot /host

In the host's journalctl output, kubelet was looping on container termination attempts. Checking the process table on the host revealed multiple kernel worker (kworker) threads in uninterruptible sleep (D state):

1ps -eo state,pid,ppid,wchan:40,comm | awk '$1 ~ /D/'

The application manifests revealed that the pod had mounted an SMB file share directly from Azure Files rather than mounting via NFS through an OpenShift CSI StorageClass. At the same time, Azure Storage had triggered an unscheduled maintenance window. While other applications mounted against Azure via NFS recovered transparently, this application remained stuck.

What is D state?

In Linux, process state D (TASK_UNINTERRUPTIBLE) represents a sleep state where a thread is waiting on a hardware event or a kernel-level resource (typically synchronous disk or network filesystem I/O) and will not process incoming signals until the call returns.

Unlike processes in interruptible sleep (S state), a process in D state ignores signals, including SIGKILL (kill -9). The process entry remains in the process table, and the thread cannot terminate until the kernel driver or system call finishes waiting or times out.

This is distinct from a zombie process (Z state). A zombie process has already terminated its execution and released its memory, remaining in the process table only until its parent reads its exit code with waitpid(). A process in D state has not finished; it is paused mid-execution inside the kernel.

What are KWorker threads?

Kernel worker threads (kworker) handle asynchronous and background tasks for the Linux kernel via kernel workqueues.

Examples include:

• Flushing dirty filesystem buffers.
• Running delayed device-driver routines.
• Processing network interface state changes.
• Handling background socket and filesystem reconnection logic.

You can inspect active kworker threads with:

1ps -eLf | grep kworker

To view their specific process states and wait channels:

1ps -eo state,pid,ppid,wchan:40,comm | grep kworker

Seeing kworker threads is normal on any Linux host. When multiple kworker threads enter D state indefinitely, it indicates that a kernel subsystem or filesystem driver is blocked waiting for an external response that has not arrived.

Why the application became stuck

The root cause was how the application mounted storage. Other applications on the cluster used NFS shares managed by OpenShift StorageClasses. This application used a direct SMB/CIFS mount configured outside the standard OpenShift CSI provisioning path.

1       OpenShift Worker Node (RHCOS)2                    │3             Application Pod4                    │5            Mounted Directory6                    │7             Host-Level Mount8                    │9            CIFS / SMB Client10                    │11           Kernel Network Stack12                    │13             Azure Files SMB

When Azure Storage underwent maintenance, the SMB endpoint stopped responding. The Linux CIFS kernel module blocked I/O requests while awaiting an SMB response packet. Because the CIFS client performs synchronous operations in uninterruptible sleep, application threads performing filesystem access, as well as kernel worker threads handling filesystem maintenance, entered D state.

Inspecting /proc/<pid>/stack for the stuck processes showed wait_for_completion in the call trace. The thread had dispatched a request to the CIFS module and suspended itself in kernel space waiting for a response that never arrived.

Why other applications were unaffected

Other workloads using the same Azure Storage account were mounted via NFS rather than SMB.

The Linux NFS and CIFS clients implement distinct timeout, reconnection, and error-handling mechanisms inside the kernel:

1. Protocol behaviors: NFS mounts typically rely on configurable hard or soft timeout parameters and RPC layer retries. CIFS maintains stateful TCP/SMB session negotiations that handle backend network resets differently during unplanned drops.
2. Mount architecture: The NFS shares were provisioned through the standard OpenShift CSI driver, integrating with the Kubernetes volume lifecycle. The SMB mount was established at the host level, bypassing CSI volume health checks, graceful detach logic, and standard Kubernetes node unmount handling.

Why OpenShift could not terminate the pods

When kubelet attempts to delete a pod:

1. Kubelet sends a stop request to the container runtime (such as CRI-O).
2. The runtime issues SIGTERM to the container's PID 1, followed by SIGKILL after the grace period expires.
3. The process in D state cannot process SIGTERM or SIGKILL because it is blocked in kernel execution.
4. The container runtime waits indefinitely for the process to exit, preventing the container cgroup from tearing down.
5. Kubelet reports the pod in Terminating or Running with persistent warning events because the container process never exited.

How to debug D-state processes on worker nodes

When a pod is stuck terminating and container logs are unreachable, use the following commands on the affected worker node.

1. Identify processes in D state

1ps -eo state,pid,ppid,wchan:40,comm | awk '$1 ~ /D/'

The wchan column displays the specific kernel function where the process is sleeping, such as wait_for_completion or cifs_reconnect.

2. Inspect the kernel call stack

1cat /proc/<PID>/stack

Look for the specific filesystem or network subsystems in the trace:

• cifs_* or smb* (CIFS/SMB filesystem calls)
• nfs_* or rpc_* (NFS and SunRPC calls)
• sock_* or tcp_* (network socket waits)
• wait_for_completion (synchronous synchronization primitives)

3. Inspect CIFS statistics and connection status

1cat /proc/fs/cifs/Stats2cat /proc/fs/cifs/DebugData

These show open SMB sessions, active socket states, pending mid requests, and reconnect attempt counters.

4. Check kernel dmesg and journalctl for storage timeouts

1journalctl -k --since "2 hours ago" | grep -iE 'cifs|smb|nfs|timeout|blocked for more than'

Look for "blocked for more than 120 seconds" warnings, which Linux emits when a process remains in D state for extended periods.

Recovery

Our priority was to bring the application back online.

A node reboot was one possible way to clear the stuck kernel operations. However, this was not an ideal solution. Other application pods were also running on the affected worker node, and rebooting it would have disrupted those workloads as well.

We decided to take a different approach.

We focused on moving the workload away from the direct host-level SMB mount and towards a CSI-managed Azure Files volume. We updated the application manifests to use PersistentVolumeClaims backed by an Azure Files StorageClass. We then force-deleted the affected pods so that Kubernetes could recreate them using the new storage configuration. The application was back online, and we did not lose any data.

However, there was still one problem.

Kubernetes events continued to report errors while trying to terminate the old pods. The reason was that some of the processes were still stuck in D state at the worker-node level. Changing the storage configuration and recreating the pods restored the application, but it did not clear the existing kernel-level blocked tasks.

The old processes were still waiting inside the kernel, and Kubernetes could not force them to exit.

At this point, a node reboot was the only practical way we identified to clear the remaining stuck processes. We decided to perform this during the next cluster upgrade to avoid an additional disruption.

Why use a StorageClass and CSI driver instead of a direct host mount?

One of the main lessons from this incident was the importance of managing storage through Kubernetes rather than creating dependencies directly on worker nodes. A StorageClass provides a declarative way to define storage provisioning requirements. The CSI driver handles the actual storage operations.

Here are some of the key benefits.

1. Kubernetes manages the storage lifecycle

With CSI, Kubernetes manages the volume lifecycle through PersistentVolumes, PersistentVolumeClaims, and the CSI interface. Operations such as volume provisioning, mounting, and unmounting follow a standard Kubernetes workflow. With a direct host mount, you have to manage more of this lifecycle outside Kubernetes.

2. Better integration with Kubernetes

CSI drivers integrate with kubelet and the Kubernetes storage subsystem. This gives Kubernetes a standard way to request storage operations and manage volumes across worker nodes. It also makes storage configuration easier to maintain across different workloads and environments.

3. Centralised storage configuration

Instead of maintaining mount commands and filesystem settings directly on worker nodes, you can define storage requirements through Kubernetes resources.

For example, a StorageClass can define:

• Storage account and file share configuration.
• SMB protocol version.
• Mount options.
• Secret references.
• Provisioning parameters.

This makes the configuration easier to review, version, and maintain.

4. Better separation between applications and infrastructure

Applications should not need to know how the underlying storage is mounted on a worker node. They should request storage through a PersistentVolumeClaim. The CSI driver and Kubernetes handle the integration with the storage backend. This reduces application dependence on specific worker-node configurations.

5. Standardised mount and unmount operations

The CSI driver handles volume mount and unmount requests through the CSI interface. Kubelet manages these operations as part of the pod lifecycle and can retry certain failed operations. This provides a more standard recovery and error-handling path than manually managed host mounts.

However, it is important to understand that CSI does not eliminate kernel-level I/O failures. If an SMB operation becomes stuck in an uninterruptible kernel wait, a CSI driver cannot necessarily recover it.

6. Easier troubleshooting and observability

With CSI-managed storage, you have additional components to investigate:

• CSI controller pods.
• CSI node plugin pods.
• Kubelet volume operation logs.
• PersistentVolume and PersistentVolumeClaim events.
• StorageClass configuration.

This gives you a more structured troubleshooting path.

One important distinction: StorageClass vs hostPath

A StorageClass and hostPath serve different purposes.

A StorageClass defines how storage is provisioned. A PersistentVolumeClaim requests that storage.

hostPath exposes a directory from the worker node's filesystem to a pod.

For example, if you manually mount an Azure Files SMB share on the worker node and then expose that directory through hostPath, Kubernetes does not manage the underlying SMB connection.

The pod only sees a directory on the host. This is the architectural difference we wanted to address. Moving to an Azure Files CSI-managed volume removes the need to maintain that direct host-level mount as part of the application deployment. It does not guarantee that SMB interruptions will never happen, but it gives Kubernetes a standard way to manage the storage lifecycle.

Takeaways

1. A `Running` pod status does not guarantee container health. A pod may report Running while its main execution threads are halted in uninterruptible kernel sleep.
2. `SIGKILL` cannot stop a `D` state process. Forcing pod termination with --grace-period=0 --force will clean up the Kubernetes API object, but the underlying host process remains running on the worker node until the I/O times out or the node is rebooted.
3. Mount architectures dictate failure modes. Direct host-level mounts bypass CSI health lifecycle management. Workloads should use standard CSI drivers and tested mount options (soft vs hard timeouts) matching their availability requirements.
4. Inspect the node layer when container logs are silent. When container logs and Kubernetes events provide no error details, host process tables (ps), wait channels (wchan), and /proc/<pid>/stack reveal whether the failure originates inside the kernel or storage network.