🧠 Section 3 · Question #20

What types of OutOfMemoryError exist

An OutOfMemoryError (OOME) in Java is not a single generic exception; it is a family of runtime errors where the JVM's error message explicitly identifies which specific memory...

🧠 Memory and Garbage Collection 📄 Source Markdown

🟢 Junior Level

An OutOfMemoryError (OOME) in Java is not a single generic exception; it is a family of runtime errors where the JVM’s error message explicitly identifies which specific memory subsystem or operating system boundary was exhausted.

Real-World Analogy

Think of a busy restaurant kitchen running out of resources:

  • Java heap space: The kitchen counters are completely covered in dirty dishes; no more plates can be prepped.
  • Metaspace: The chef’s recipe binder is out of pages; no new recipes or menu items can be written down.
  • Direct buffer memory: The walk-in refrigerator leased from the landlord next door (Off-Heap Native Memory) is full, even though the main kitchen counters are empty.
  • unable to create new native thread: The city fire marshal limits the building to 50 staff members; trying to hire line cook #51 gets the restaurant shut down (OS Thread Limit).

Memory Subsystems & Their Associated OOMEs

┌────────────────────────────────────────────────────────────────────────┐
│                        JVM MEMORY SUBSYSTEMS                           │
├────────────────────────────────────────────────────────────────────────┤
│ 1. JAVA HEAP (-Xmx)                                                    │
│    • `Java heap space` (Heap full of unreachable/retained objects)     │
│    • `GC overhead limit exceeded` (GC spends >98% CPU freeing <2% RAM) │
│    • `Requested array size exceeds VM limit` (Array > Integer.MAX_VAL) │
├────────────────────────────────────────────────────────────────────────┤
│ 2. METASPACE & CLASS METADATA (Native Off-Heap)                        │
│    • `Metaspace` (Class metadata exceeds -XX:MaxMetaspaceSize)         │
│    • `Compressed class space` (Exceeds -XX:CompressedClassSpaceSize)   │
├────────────────────────────────────────────────────────────────────────┤
│ 3. OPERATING SYSTEM / NATIVE MEMORY (Off-Heap)                         │
│    • `Direct buffer memory` (NIO/Netty exceeds -XX:MaxDirectMemorySize)│
│    • `unable to create new native thread` (OS ulimit / RAM exhausted)  │
│    • `request <size> bytes for <reason>. Out of swap space?` (malloc)  │
│    • `Map failed` (Linux vm.max_map_count limit reached)               │
└────────────────────────────────────────────────────────────────────────┘

🟡 Middle Level

The 5 Most Frequent Production OOMEs: Root Causes & Solutions

1. java.lang.OutOfMemoryError: Java heap space

  • Root Cause: The volume of live objects reachable from active GC Roots exceeds the maximum configured heap limit (-Xmx).
  • Common Triggers: Unbounded static collections, unevicted in-memory caches, memory leaks in session handlers, or an undersized heap under sudden traffic bursts.
  • Remediation: Capture an automatic heap dump (-XX:+HeapDumpOnOutOfMemoryError) and inspect the Dominator Tree in Eclipse MAT to locate the root memory accumulator.

2. java.lang.OutOfMemoryError: GC overhead limit exceeded

  • Root Cause: A built-in HotSpot safety heuristic that triggers when the JVM spends $> 98\%$ of total CPU time executing GC pauses while reclaiming $< 2\%$ of heap space.
  • Why it Exists: Prevents the application from locking up indefinitely in a near-permanent Stop-The-World loop with zero throughput.
  • Remediation: Do not disable the heuristic via -XX:-UseGCOverheadLimit! Fix the underlying heap leak or increase heap headroom (-Xmx).

3. java.lang.OutOfMemoryError: Metaspace vs. Compressed class space

  • Metaspace: Loaded class metadata, method bytecodes, constant pools, and annotations exceed -XX:MaxMetaspaceSize.
    • Trigger: ClassLoader leaks during dynamic application redeployments or un-cached runtime Groovy/JavaScript compilations.
  • Compressed class space: Exceeds the 1 GB dedicated region reserved for 32-bit InstanceKlass pointers (-XX:CompressedClassSpaceSize=1g).
    • Trigger: Generating millions of dynamic proxy classes (CGLIB, ByteBuddy, Spring AOP) without caching.

4. java.lang.OutOfMemoryError: Direct buffer memory

  • Root Cause: Off-heap memory allocated via ByteBuffer.allocateDirect() (frequently used by Netty, gRPC, and Kafka) exceeds -XX:MaxDirectMemorySize (which defaults to -Xmx).
  • The Paradox: Can occur even when the Java Heap is 95% empty!
  • Remediation: Ensure byte buffers are returned to buffer pools (ReferenceCountUtil.release(buf)) and track allocations using Native Memory Tracking (-XX:NativeMemoryTracking=detail).

5. java.lang.OutOfMemoryError: unable to create new native thread

  • Root Cause: The JVM attempted to spawn a new platform thread (new Thread()), but the host operating system refused the allocation.
  • The Culprits:
    1. The OS kernel process thread limit was breached (Linux ulimit -u or /proc/sys/kernel/pid_max).
    2. Physical RAM is exhausted by thread stacks ($N_{\text{threads}} \times \text{-Xss1m}$).
  • Remediation: Check ulimit -u, eliminate unbounded thread pools, reduce stack size (-Xss512k), or migrate blocking I/O to Java 21 Virtual Threads.

🔴 Senior Level

Advanced & Edge-Case OOME Types

1. Requested array size exceeds VM limit

// Throws: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
int[] massiveArray = new int[Integer.MAX_VALUE];
  • Why it Happens: In the HotSpot JVM, an array object header stores its length in a 32-bit signed integer. However, the JVM reserves a few header words for internal metadata (object header, mark word, klass pointer).
  • Consequently, the maximum allowable array size is between Integer.MAX_VALUE - 2 and Integer.MAX_VALUE - 8 (typically $2{,}147{,}483{,}639$ elements), regardless of available RAM.

2. request <size> bytes for <reason>. Out of swap space?

  • Mechanism: The JVM’s internal C++ runtime attempted to allocate native memory via os::malloc() and received a NULL return pointer.
  • Indication: The host operating system has run completely out of physical RAM and swap space.
  • Output: The JVM immediately aborts execution and generates a fatal native crash log (hs_err_pid.log).

3. Map failed

  • Mechanism: Occurs when using memory-mapped I/O (FileChannel.map() / MappedByteBuffer) and the operating system rejects the mmap() system call.
  • Linux Cause: The process breached the maximum virtual memory map count enforced by the Linux kernel (sysctl vm.max_map_count, default 65,530). Common in Elasticsearch, Kafka, and Lucene databases.
  • Remediation: Enlarge kernel mapping limits: sysctl -w vm.max_map_count=262144.

Diagnostic Decision Matrix

Error Detail Message First Diagnostic Command Primary JVM / OS Tuning Flags
Java heap space Open .hprof dump in Eclipse MAT -Xmx, -XX:MaxRAMPercentage
GC overhead limit exceeded Inspect GC logs (-Xlog:gc*) for thrashing Profile object churn, increase -Xmx
Metaspace jcmd <PID> VM.metaspace show-loaders -XX:MaxMetaspaceSize=512m
Compressed class space Inspect proxy generation in classloaders -XX:CompressedClassSpaceSize=2g
Direct buffer memory jcmd <PID> VM.native_memory detail -XX:MaxDirectMemorySize=2g
unable to create native thread Check ulimit -u and jstack thread counts -Xss512k, use Virtual Threads (Java 21)
Map failed Check cat /proc/sys/vm/max_map_count sysctl -w vm.max_map_count=262144

4 Tricky Questions

1. Why does increasing -Xmx actually make OutOfMemoryError: unable to create new native thread worse rather than better?

Answer: The unable to create new native thread error occurs when the operating system refuses to allocate memory for a new OS thread stack (each platform thread allocates a dedicated 1 MB stack via -Xss1m from native OS RAM, completely outside the Java Heap). In a 64-bit container with a fixed memory ceiling (e.g., 4 GB), expanding -Xmx from 2 GB to 3.5 GB forces the JVM to claim almost all available physical RAM for the Heap. This leaves a tiny sliver of only 500 MB for thread stacks, Metaspace, and native code. With less native RAM remaining, the operating system runs out of memory faster, reducing the maximum number of threads the application can spawn and accelerating the crash. The proper fix is often to decrease -Xmx or migrate to Java 21 Virtual Threads.

2. What is the fundamental difference between OutOfMemoryError: Metaspace and OutOfMemoryError: Compressed class space?

Answer: Metaspace is the broad native memory area storing all class metadata (method bytecodes, runtime constant pools, annotations, and virtual method dispatch tables). Within Metaspace, the JVM reserves a specific, contiguous 1 GB memory block called the Compressed Class Space (-XX:CompressedClassSpaceSize=1g) when Compressed Class Pointers are enabled (-XX:+UseCompressedClassPointers). This 1 GB region stores only the C++ InstanceKlass structures so they can be referenced using 32-bit pointer offsets from object headers. An application can throw Compressed class space OOM if it generates hundreds of thousands of dynamic proxy classes, even if general Metaspace has plenty of headroom and the host OS has gigabytes of free RAM.

3. What does the JVM flag -XX:-UseGCOverheadLimit do, and why is disabling it considered an anti-pattern?

Answer: By default, HotSpot enables -XX:+UseGCOverheadLimit, a safety heuristic that throws an OutOfMemoryError if the JVM spends more than 98% of its CPU time executing Garbage Collection while recovering less than 2% of heap memory. Disabling this flag via -XX:-UseGCOverheadLimit simply turns off this safety check. It does not solve the memory problem; instead, it allows the JVM to enter a state of total CPU thrashing: all CPU cores run at 100% execution capacity executing back-to-back Stop-The-World Full GCs, application throughput drops to zero, and the application freezes indefinitely until it finally crashes minutes later with OutOfMemoryError: Java heap space.

4. How can OutOfMemoryError: Direct buffer memory occur when the Java Heap is 95% empty?

Answer: Direct memory is allocated outside the Java Heap in native operating system memory via ByteBuffer.allocateDirect() (using native C malloc()). On the Java Heap, a DirectByteBuffer is represented only by a tiny Java wrapper object occupying roughly 64 bytes. The deallocation of the underlying native memory is handled by a Cleaner (based on PhantomReference) that executes only when the 64-byte wrapper object is garbage collected. If the application performs heavy network I/O (allocating gigabytes of native direct buffers) but creates very few on-heap objects, the Java Heap never fills up. Because the heap is 95% empty, the Garbage Collector never triggers, the Java wrapper objects are never collected, and the underlying native direct memory grows unchecked until it breaches -XX:MaxDirectMemorySize.

🎯 Interview Cheat Sheet

30-Second Elevator Pitch

“An OutOfMemoryError indicates exhaustion of a specific JVM memory subsystem, not just the Heap. While Java heap space signals that live objects exceed -Xmx, GC overhead limit exceeded warns that the collector is thrashing (>98% CPU spent freeing <2% memory). In native memory, Metaspace and Compressed class space occur when dynamic bytecode generation exhausts class metadata areas. Direct buffer memory strikes when off-heap NIO buffers exceed -XX:MaxDirectMemorySize (even on an empty heap). Finally, unable to create new native thread occurs when OS thread limits or native stack memory are exhausted; counter-intuitively, increasing -Xmx worsens this error by starving native RAM.”

Key Architectural Takeaways

  • Specific Error Messages: Never treat OOMEs generically; the detail message pinpoints the exact exhausted subsystem.
  • Heap vs. Native: Java heap space and GC overhead occur in the Heap; Metaspace, Direct buffer, and native thread occur in native OS RAM.
  • GC overhead limit: A safety valve preventing infinite GC loops; never disable it with -XX:-UseGCOverheadLimit.
  • Thread Creation Paradox: Increasing -Xmx reduces native RAM available for thread stacks, worsening unable to create new native thread.
  • Array Size Limit: Arrays cannot exceed Integer.MAX_VALUE - 2 due to 32-bit length header constraints.
  • Direct Buffer Decoupling: Direct buffers reside off-heap; they can exhaust native memory even when the heap is nearly empty.

Red Flags & Anti-Patterns

  • 🚩 Assuming every OOME is solved by increasing -Xmx: Increasing -Xmx accelerates native thread and direct buffer crashes.
  • 🚩 Disabling -XX:-UseGCOverheadLimit to “fix” an error: Simply trades a fast failure for a total CPU freeze followed by a crash.
  • 🚩 Confusing Metaspace with Compressed class space: Fails to recognize the 1 GB ceiling reserved for 32-bit Klass pointers.
  • 🚩 Confusing StackOverflowError with unable to create new native thread: StackOverflow is deep recursion on one thread; native thread OOM is failure to spawn a new OS thread.
  • 🚩 Ignoring Linux kernel limits (vm.max_map_count): Fails to diagnose Map failed errors in database and messaging workloads.