🧠 Section 3 · Question #22

What tools help analyze memory

Diagnosing memory consumption, garbage collection overhead, and memory leaks in Java requires a specialized suite of tools ranging from standard JDK command-line utilities to gr...

🧠 Memory and Garbage Collection 📄 Source Markdown

🟢 Junior Level

Diagnosing memory consumption, garbage collection overhead, and memory leaks in Java requires a specialized suite of tools ranging from standard JDK command-line utilities to graphical dump analyzers and low-overhead production profilers.

Modern Memory Toolset Taxonomy

Tool Category Primary Use Case Production Overhead
Eclipse MAT Desktop GUI Deep offline post-mortem analysis of HPROF heap dumps 0% (Analyzes dumps offline)
jcmd JDK CLI Universal command-line diagnostic interface for HotSpot JVM Negligible (Microsecond command execution)
JFR / JMC JDK Core Profiler Continuous telemetry, allocation profiling, leak sampling < 1% CPU (Engineered for production)
VisualVM Desktop GUI Live heap, thread, CPU monitoring, lightweight sampling Low–Medium (Development and QA environments)
Async-profiler Native Profiler CPU, allocation flame graphs, native memory tracing without safepoint bias < 2% CPU (Production-safe)
JProfiler / YourKit Commercial GUI Interactive allocation profiling and deep object graphs High (20–100%) (BCI mode; dev/test only)

Core Desktop Analyzer: Eclipse MAT

Eclipse Memory Analyzer (MAT) is the industry-standard open-source tool for deep inspection of .hprof binary heap dumps:

  • Dominator Tree: Ranks objects by Retained Heap (the total memory freed if that object and its exclusively retained sub-tree were garbage collected).
  • Leak Suspects Report: Automatically executes heuristic checks to pinpoint oversized collections, saturated caches, or massive thread structures.
  • Path to GC Roots: Traces the exact reference chain keeping an object alive, allowing you to exclude soft, weak, or phantom references to identify strong reference leaks.
  • Object Query Language (OQL): An SQL-like syntax for querying the in-memory object graph (e.g., SELECT * FROM java.lang.String s WHERE s.count > 1000).

🟡 Middle Level

1. The Modern CLI Standard: jcmd

Historically, developers relied on disparate tools like jmap, jstack, and jinfo. In modern HotSpot distributions, jcmd is the officially recommended, unified utility that communicates directly with the JVM via the HotSpot Attach API.

# List all running JVM processes on the host
jcmd -l

# 1. Quick in-memory class histogram (fast overview of top instance types)
jcmd <pid> GC.class_histogram

# 2. Trigger an immediate HPROF heap dump to disk
jcmd <pid> GC.heap_dump /var/dumps/heap_analysis.hprof

# 3. Trigger a heap dump containing ONLY reachable (live) objects (forces a full GC first)
jcmd <pid> GC.heap_dump -all=false /var/dumps/live_dump.hprof

# 4. View Metaspace summary and class loader hierarchies
jcmd <pid> VM.metaspace show-loaders

# 5. Inspect Native Memory Tracking (NMT)
jcmd <pid> VM.native_memory summary

[!TIP] Why run GC.class_histogram before GC.heap_dump? A class histogram scans object headers across the heap and aggregates instance counts and shallow bytes during a sub-second Safepoint. It does not serialize references or write gigabytes to disk, making it a fast and safe diagnostic first step on a live production node.

2. Live GC Telemetry: jstat

For real-time terminal monitoring of generational space utilization and garbage collection pauses without attaching a heavy profiler:

# Sample GC statistics every 1000ms for process <pid>
jstat -gcutil <pid> 1000

Sample output:

  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT    CGC    CGCT     GCT   
  0.00 100.00  42.50  88.20  96.40  92.10     42    0.840     2    0.310     6    0.045    1.195
  • S0 / S1: Survivor 0 and Survivor 1 occupancy (%).
  • E / O: Eden and Old generation occupancy (%).
  • M / CCS: Metaspace and Compressed Class Space occupancy (%).
  • YGC / YGCT: Young GC cycle count and total accumulated time (seconds).
  • FGC / FGCT: Full GC cycle count and total accumulated pause time (seconds).

3. Java Flight Recorder (JFR) & JDK Mission Control (JMC)

JFR is an event recorder built directly into the HotSpot JVM kernel. It records VM internal events into lock-free native memory ring buffers with less than 1% CPU overhead.

# Launch a 15-minute continuous production recording with the 'profile' preset
jcmd <pid> JFR.start name=ProdProfile settings=profile duration=15m filename=/var/log/app_profile.jfr

# Check active recording status
jcmd <pid> JFR.check

# Dump the current in-flight ring buffer on demand
jcmd <pid> JFR.dump name=ProdProfile filename=/var/log/snapshot.jfr

Key memory diagnostic capabilities in JFR / JMC:

  • jdk.OldObjectSample: Samples long-lived objects that survive generational promotions, providing full allocation stack traces and reference chains to GC Roots without requiring a heap dump.
  • Allocation Profiling: Measures allocation rates inside TLAB (Thread Local Allocation Buffer) and outside TLAB (direct-to-Old allocations), isolating high-allocation call stacks that induce GC pressure.

🔴 Senior Level

1. Off-Heap Diagnosis: Native Memory Tracking (NMT)

When process Resident Set Size (RSS) grows continuously toward container memory limits (limits.memory), but Java Heap metrics remain low and stable, the leak exists in Native Memory (Off-Heap).

Enable NMT at JVM startup:

# Options: 'summary' (~1% CPU overhead) or 'detail' (5-10% CPU overhead due to malloc call-site wrapping)
java -XX:NativeMemoryTracking=detail -jar application.jar

Differential Baseline Workflow

# Step 1: Establish baseline after initial startup and warm-up
jcmd <pid> VM.native_memory baseline

# Step 2: Run workload or wait for memory degradation

# Step 3: Compare growth against the baseline
jcmd <pid> VM.native_memory detail.diff

Sample differential output:

Total: reserved=5840MB +180MB, committed=3410MB +125MB
-                 Java Heap (reserved=4096MB, committed=2048MB)
-                     Class (reserved=1085MB +5MB, committed=65MB +3MB)
-                    Thread (reserved=140MB +40MB, committed=140MB +40MB)  <-- Leak: +40 unpooled threads!
-                      Code (reserved=248MB +2MB, committed=45MB +1MB)
-                        GC (reserved=180MB, committed=180MB)
-             Internal(NIO) (reserved=350MB +80MB, committed=350MB +80MB) <-- Leak: DirectByteBuffer leak!

2. Async-Profiler for Allocation Flame Graphs

Traditional Java profilers utilize Bytecode Instrumentation (BCI) or JVMTI GetStackTrace, which suffer from Safepoint Bias (samples are only collected when threads reach safepoints, skewing measurements away from uncounted loops or native operations).

Async-profiler bypasses safepoints by intercepting HotSpot TLAB allocation hooks (InlineCacheBuffer, OptoRuntime::new_instance_C) and OS kernel perf_events:

# Capture 60 seconds of heap allocations and output an interactive FlameGraph:
./asprof -d 60 -e alloc -f alloc_flamegraph.html <pid>

# Profile native C/C++ memory allocations (glibc malloc):
./asprof -d 60 -e malloc -f native_malloc_flamegraph.html <pid>

+-------------------------------------------------------------------------+
|                  Allocation Flame Graph Visualization                   |
+-------------------------------------------------------------------------+
|  [ com.example.service.OrderService.processOrder() (1.2 GB / 68%) ]     |
|    [ com.fasterxml.jackson.databind.ObjectMapper.readValue() (850 MB) ]|
|      [ byte[] (alloc in TLAB) (600 MB) ]  [ String (250 MB) ]          |
+-------------------------------------------------------------------------+

3. Hot Diagnostics with Alibaba Arthas

In containerized or air-gapped environments where extracting multi-gigabyte files or restarting JVMs with new agent flags is prohibited, Arthas attaches to running JVMs on demand:

# Launch interactive Arthas CLI
java -jar arthas-boot.jar

# Inside Arthas interactive terminal:
[arthas]$ dashboard          # Real-time dashboard of Heap, Metaspace, GC count, and top threads
[arthas]$ memory             # Fine-grained memory pool breakdown
[arthas]$ vmtool --action getInstances --className com.example.HeavyCache --limit 1
[arthas]$ heapdump --live /tmp/live_dump.hprof  # Trigger live-only dump directly

4. Production Pitfalls and Failure Modes

  1. Heap Dumps on Huge Heaps ($\ge 32\text{ GB}$): Taking a full heap dump forces a global Safepoint while writing the serialized object graph to disk. On spinning disks or saturated cloud volumes, a 32 GB heap can freeze the JVM for 30–90 seconds, causing health-check timeouts, pod evictions, and circuit breaker cascades.
  2. NMT detail in Production: Tracking every native malloc call site incurs significant metadata overhead and up to 10% CPU penalty. Use -XX:NativeMemoryTracking=summary in production and reserve =detail for diagnostic environments.
  3. Destructive CLI Flags (-F): Deprecated utilities like jmap -F or jstack -F attach via OS-level ptrace. If the process is under heavy load or locks, ptrace can permanently hang or segfault the target JVM.

4 Tricky Questions

1. Why is jcmd <pid> GC.class_histogram drastically safer in production than jcmd <pid> GC.heap_dump?

Answer: GC.class_histogram iterates over object headers within committed memory blocks to tally instance counts and byte sizes per Klass*. It performs no graph traversal, reference serialization, or heavy disk I/O. As a result, the Stop-The-World (STW) pause is typically between 50 ms and 500 ms even on multi-gigabyte heaps.

In contrast, GC.heap_dump must freeze the JVM (STW) and traverse every single object reference, primitive field value, and stack frame to write an HPROF binary file to disk. Writing 16–64 GB across I/O-bound disks can stall application threads for tens of seconds, causing upstream network timeouts, broken database connection pools, and container termination by orchestrator health probes.

2. How do you diagnose a memory leak when a heap dump shows 80% free heap, but the Linux container RSS is crashing with Exit Code 137 (OOM Killer)?

Answer: An empty Java heap combined with ballooning process RSS indicates a Native Off-Heap Memory Leak. Because standard HPROF heap dumps only capture on-heap Java objects, off-heap native memory is invisible to tools like Eclipse MAT.

Diagnostic Procedure:

  1. Enable Native Memory Tracking: -XX:NativeMemoryTracking=summary.
  2. Generate baseline and diff via jcmd <pid> VM.native_memory baseline and jcmd <pid> VM.native_memory detail.diff.
  3. Check key suspects:
    • Thread section: Unbounded thread spawning (each thread consumes $\sim 1\text{ MB}$ stack off-heap).
    • Internal section: Leaked Direct ByteBuffers (e.g., Netty ByteBufs not released via ReferenceCountUtil.release()).
    • Class section: Metaspace growth due to dynamic class generation (CGLIB/ByteBuddy) without class loader unloading.
    • Arena Chunk / C-Heap: Native memory allocated directly by JNI/C++ libraries via malloc() without freeing (diagnosed using asprof -e malloc or jemalloc memory profiling).

3. How does JFR’s jdk.OldObjectSample event identify memory leaks without triggering a full Stop-The-World heap dump?

Answer: JFR maintains an internal low-overhead ring buffer of sample objects (typically 256 to 512 entries) allocated throughout application runtime. When an object is allocated, JFR captures its allocation stack trace and registers a weak handle to the instance.

As minor GCs occur, short-lived objects die and their sample entries are recycled. If a sampled object survives repeated GC cycles and is promoted to the Old Generation, JFR traces the reference path from the object back toward GC Roots during regular garbage collection phases. In JDK Mission Control (JMC), developers can inspect the live reference chain and the exact line of code where the leaked object was originally instantiated—all captured with sub-1% runtime overhead and zero custom STW pauses.

4. Why can profiling memory allocations using bytecode instrumentation (BCI) cause severe performance degradation and skewed results compared to async-profiler or JFR?

Answer: Bytecode instrumentation (used by classic GUI profilers when allocation tracing is enabled) injects callback bytecode instructions (e.g., ProfilerTracker.onAlloc(this)) into every class constructor and allocation call site across the application:

  1. Inlining Destruction: Injected callback logic increases bytecode size past JIT compiler inlining thresholds (-XX:MaxInlineSize), preventing C2 from performing critical compiler optimizations like scalar replacement and loop unrolling.
  2. Locking & Synchronization: Instrumentation listeners must synchronize or write to shared data structures across thousands of concurrent threads, creating artificial contention and cache invalidations.
  3. Distorted Allocation Rates: The profiler itself generates extra objects and method frames during measurement, altering the garbage collection dynamics it intends to measure.

In contrast, JFR and async-profiler sample allocations asynchronously inside HotSpot’s internal TLAB acquisition routines (tlab_end / tlab_top boundary refills) without touching application bytecode.

🎯 Interview Cheat Sheet

30-Second Summary

“Java memory analysis relies on three primary tiers:

  1. HotSpot CLI tools (jcmd, jstat): For fast, on-server telemetry such as class histograms (GC.class_histogram) and Native Memory Tracking (VM.native_memory) with sub-second impact.
  2. Continuous profilers (JFR/JMC and Async-profiler): Engineered for production use with $<1\%$ overhead; JFR uses jdk.OldObjectSample to locate memory leaks without STW pauses, while async-profiler generates allocation FlameGraphs without safepoint bias.
  3. Offline dump analyzers (Eclipse MAT): For post-mortem investigation of HPROF dumps generated after an OOME, computing the Dominator Tree and Retained Heap to isolate the root cause.”

Common Follow-up Questions

  • “What is the difference between Shallow Size and Retained Size in MAT?” → Shallow size is the memory consumed by the object itself (header + fields). Retained size is the shallow size plus the size of all objects that are exclusively reachable from this object (the memory reclaimed if this object is deleted).
  • “Can you inspect DirectByteBuffers in a heap dump?” → You can see the java.nio.DirectByteBuffer Java object ($64\text{ bytes}$ on-heap), its capacity, and its address pointer, but the backing native memory buffer ($100\text{ MB}$) is allocated outside the JVM heap and cannot be inspected directly in HPROF dumps.
  • “What happens if you run jcmd <pid> GC.heap_dump -all=false?” → -all=false forces a Full Garbage Collection before writing the dump, ensuring that only actively reachable (live) objects are included in the file.

Red Flags (DO NOT Say)

  • ❌ “I use jmap -dump:live in production because it’s the newest tool.” (jmap is legacy; jcmd is the HotSpot standard).
  • ❌ “A Heap Dump captures all memory used by the operating system process.” (HPROF dumps only capture the Java Object Heap; Metaspace, thread stacks, and off-heap native memory are omitted).
  • ❌ “Running JProfiler with allocation recording enabled in production is safe.” (BCI instrumentation introduces 20–100% overhead, disrupts JIT optimizations, and risks destabilizing production services).