---

🟢 Junior Level

A Heap Dump can be captured through three primary operational channels: automatically on process failure, manually on-demand via JDK command-line tools, or interactively through graphical developer environments.

The 3 Core Acquisition Methods

# 1. Automatic Post-Mortem Capture on OutOfMemoryError (Production Essential)
java -XX:+HeapDumpOnOutOfMemoryError \
     -XX:HeapDumpPath=/var/log/dumps/app_oom.hprof \
     -jar microservice.jar

# 2. Manual On-Demand Capture via jcmd (Modern JDK Standard)
# Step A: Identify Java Process ID (PID)
jps -v
# Step B: Trigger heap dump
jcmd <PID> GC.heap_dump /tmp/heap_analysis.hprof

# 3. Graphical Developer IDE / Tooling (Development & Staging)
# VisualVM: Right-click target process -> "Heap Dump"
# IntelliJ IDEA Ultimate: Profiler Tool Window -> "Capture Memory Snapshot"

---

🟡 Middle Level

1. Advanced CLI Parameters with jcmd

The jcmd tool provides flags to control disk overhead and garbage filtering:

# On-the-fly streaming Gzip compression (Java 11u12+ and Java 17+)
# -gz=1 provides ultra-fast compression with minimal CPU overhead
# -gz=9 provides maximum compression ratio (smaller file, higher CPU)
jcmd <PID> GC.heap_dump -gz=1 /tmp/dump.hprof.gz

# Capture ONLY reachable (live) objects (forces a Stop-The-World Full GC first)
jcmd <PID> GC.heap_dump -all=false /tmp/live_dump.hprof

[!NOTE] Compressing a 32 GB heap dump with -gz=1 reduces the file to approximately 6–8 GB, significantly reducing disk I/O bottlenecks and file transfer latency across remote servers.

---

2. Programmatic Capture via HotSpotDiagnosticMXBean

For custom operational tooling or self-diagnostic maintenance tasks, capture dumps directly within Java application code:

package com.example.diagnostics;

import com.sun.management.HotSpotDiagnosticMXBean;
import java.io.IOException;
import java.lang.management.ManagementFactory;

public class HeapDumpService {

    public static void dumpHeap(String destinationPath, boolean liveOnly) throws IOException {
        HotSpotDiagnosticMXBean mxBean = ManagementFactory.newPlatformMXBeanProxy(
                ManagementFactory.getPlatformMBeanServer(),
                "com.sun.management:type=HotSpotDiagnostic",
                HotSpotDiagnosticMXBean.class
        );
        // liveOnly=true triggers a synchronous Full GC before dump serialization
        mxBean.dumpHeap(destinationPath, liveOnly);
    }
}

---

3. Spring Boot Actuator Endpoint

Spring Boot provides an out-of-the-box management endpoint:

# Stream heap dump directly over HTTP
curl -O http://localhost:8080/actuator/heapdump

• Internally delegates to HotSpotDiagnosticMXBean.dumpHeap(), saves a temporary file on the host, and streams it back as application/octet-stream.
• ⚠️ Critical Security Warning: This endpoint must be strictly restricted to internal networks and protected by administrative role-based access control (RBAC). Exposing it publicly leaks all plaintext memory contents (tokens, database passwords, PII).

---

🔴 Senior Level

1. HotSpot Attach API Mechanics & Container Security Traps

When executing jcmd <PID> GC.heap_dump, the tool communicates with the target JVM via the HotSpot Attach API:

[jcmd Client Process]
   │
   ├─► 1. Creates signal file: /tmp/.attach_pid<PID>
   ├─► 2. Sends OS signal: SIGQUIT (3) to target JVM process
   │
[Target JVM Process]
   │
   ├─► 3. Signal Dispatcher thread spawns Attach Listener thread
   ├─► 4. Attach Listener opens Unix Domain Socket: /tmp/.java_pid<PID>
   │
[jcmd Client Process]
   │
   └─► 5. Connects to Unix Domain Socket and sends command "GC.heap_dump <args>"

Common Container Pitfalls in Docker & Kubernetes:

• Unable to open socket file: Production containers run under an unprivileged user (e.g., appuser with UID 10001). If an engineer executes kubectl exec as root and runs jcmd 1 ..., the Linux kernel rejects the connection because the socket file /tmp/.java_pid1 belongs to UID 10001. Remedy: Execute the command under the exact UID of the running application:

  su -s /bin/sh appuser -c "jcmd 1 GC.heap_dump -gz=1 /dumps/dump.hprof.gz"
  

• Filesystem Mount Constraints: If /tmp is mounted with noexec or nosuid, or if the JVM was started with -XX:+DisableAttachMechanism, the Attach API is completely disabled.

---

2. Live Kubernetes Pod Dump Workflow (Zero Downtime)

Capturing a heap dump from a live production pod without risking cascading service failures:

# Step 1: Drain traffic by removing the pod selector label
kubectl label pod order-service-7f8d-xkz pod-under-diag=true --overwrite
kubectl label pod order-service-7f8d-xkz app-

# Step 2: Trigger compressed dump to a mounted Persistent Volume (PV)
kubectl exec order-service-7f8d-xkz -- \
  su -s /bin/sh appuser -c "jcmd 1 GC.heap_dump -gz=1 /dumps/diag.hprof.gz"

# Step 3: Copy dump artifact to local engineering workstation
kubectl cp order-service-7f8d-xkz:/dumps/diag.hprof.gz ./diag.hprof.gz

# Step 4: Clean up temporary dump from persistent storage
kubectl exec order-service-7f8d-xkz -- rm /dumps/diag.hprof.gz

# Step 5: Terminate the drained diagnostic pod (orchestrator will spawn a fresh replica)
kubectl delete pod order-service-7f8d-xkz

---

3. Capturing Frozen JVMs: OS Core Dumps (gcore) & jhsdb

When a JVM is completely unresponsive (e.g., deadlocked in native code, frozen in uninterruptible disk sleep, or failing to respond to Attach API signals), jcmd hangs indefinitely. The solution is taking an OS-level core dump via the Linux kernel:

# Step 1: Capture non-invasive Linux process core dump via GDB
gcore -o /dumps/core.dump <PID>

# Step 2: Offline conversion of OS core dump into standard Java HPROF using jhsdb
jhsdb jmap --exe $JAVA_HOME/bin/java \
           --core /dumps/core.dump.<PID> \
           --binaryheap \
           --dumpfile /dumps/offline_converted.hprof

Because gcore operates at the Linux kernel level via ptrace, it captures the entire process memory space without requiring the JVM to reach a Safepoint.

---

4 Tricky Questions

1. Why does running jcmd 1 GC.heap_dump inside a Kubernetes container as root frequently fail with Unable to open socket file?

Answer: The HotSpot Attach API relies on Unix Domain Sockets located in the temporary directory (typically /tmp/.java_pid<PID>). The Linux virtual filesystem strictly enforces ownership and permission checks on domain sockets.

When a Java process runs under an unprivileged application user (e.g., UID 10001), the socket is owned by UID 10001 with permissions 0600. When an operator connects via kubectl exec -it <pod> -- sh, the interactive shell defaults to root (UID 0). Even though root possesses system-wide capabilities, the HotSpot client security validation verifies that the client process UID matches the target JVM’s UID exactly to prevent arbitrary local privilege escalation. The command fails immediately with Unable to open socket file.

---

2. What are the severe operational and security risks of exposing Spring Boot’s /actuator/heapdump endpoint?

Answer: Exposing /actuator/heapdump presents two catastrophic risks:

1. Denial of Service (DoS): Triggering the endpoint invokes HotSpotDiagnosticMXBean.dumpHeap(), forcing the JVM into a multi-second Stop-The-World pause while serializing gigabytes of data. Concurrently, streaming this data over HTTP consumes massive network bandwidth and CPU cycles. Repeated HTTP requests can easily crash the container or cause health-check failures.
2. Critical Security Breach: A heap dump contains unencrypted plaintext copies of all in-memory application data: database credentials, private encryption keys, JWT secrets, and customer PII. Anyone with network access to the endpoint can download the entire application memory state.

---

3. How do you extract an HPROF heap dump from a JVM that is completely frozen in an uninterruptible native system call or deadlock and does not respond to jcmd?

Answer: If the JVM cannot reach a Safepoint or its Attach Listener thread is unresponsive, all standard Java diagnostic tools (jcmd, jmap, VisualVM) will hang indefinitely waiting for the socket connection.

Forensic Solution:

1. Use the Linux operating system utility gcore <PID> to force the kernel to write a standard Linux ELF core dump (core.<PID>).
2. Alternatively, send SIGABRT (kill -6 <PID>) with -XX:+CreateCoredumpOnCrash enabled to force the OS to terminate the process and write a core dump.
3. Use the HotSpot Serviceability Agent utility jhsdb offline to convert the native core dump into an HPROF file:

   jhsdb jmap --exe $JAVA_HOME/bin/java --core core.<PID> --binaryheap --dumpfile crash.hprof
   

---

4. What happens inside the JVM if the disk fills up to 100% capacity (ENOSPC) while writing an HPROF dump?

Answer: During heap dump serialization, the JVM executes buffered write() system calls to stream data to the file descriptor. If the target disk runs out of space, the OS returns the ENOSPC (No space left on device) error code.

The JVM catches the underlying IOException:

1. It immediately aborts the dumping procedure and closes the file descriptor.
2. It logs an error to standard error / GC log: Unable to create <path>: No space left on device.
3. It resumes application mutator threads, exiting the Safepoint STW state so the application can attempt to continue running.
4. The .hprof file remaining on disk is incomplete and corrupted (missing terminal records and footer indices). Most GUI tools will reject it unless repaired with specialized binary truncation recovery scripts.

---

🎯 Interview Cheat Sheet

30-Second Summary

“Heap dumps are captured automatically during fatal failures using -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath, or manually on demand via jcmd <PID> GC.heap_dump [-gz=1] <path>. Streaming compression (-gz=1) is recommended to reduce file size by up to 75%. In containerized environments, dumps must be written to external Persistent Volumes rather than ephemeral container disks, and jcmd must be executed under the application’s exact UID. When JVMs are completely frozen, offline recovery using gcore and jhsdb jmap provides a kernel-level fallback.”

Common Follow-up Questions

• “How do you find the PID of a Java application in Linux?” → Use jps -v or pgrep -a java. Inside containers, Java is almost always PID 1.
• “What is the difference between -gz=1 and -gz=9 in jcmd?” → -gz=1 uses minimal deflate compression, providing the highest throughput with minimal CPU overhead; -gz=9 achieves the smallest file size but consumes significantly more CPU cycles and takes longer.
• “Can you trigger a heap dump via JMX?” → Yes, using com.sun.management.HotSpotDiagnosticMXBean.dumpHeap(path, live).

Red Flags (DO NOT Say)

• ❌ “I use jmap because jcmd is deprecated.” (jmap is legacy; jcmd is the official modern standard).
• ❌ “It is fine to expose /actuator/heapdump on public internet endpoints.” (Severe security vulnerability; leaks plaintext secrets and customer PII).
• ❌ “Taking a heap dump in a Docker container never has permissions issues.” (UID/GID mismatch between root exec and the unprivileged container user triggers Unable to open socket file).

---

Related Topics

• What is a heap dump — HPROF structure and contents
• What tools help analyze memory — Tooling overview (MAT, JMC, VisualVM)
• What is a memory leak and how to detect it — Analysis workflows
• What happens on OutOfMemoryError — Automatic crash dump dynamics
• What is stop-the-world — STW pause mechanics during dump generation