Out Of Memory – Types, Causes, Analysis

—————————————————————————————————————————————-

At any time the JVM may run out of memory, which effectively crashes the container and will require a restart to restore functionality. There are many causes for out of memory exceptions. From too small of a heap defined for the applications running in the container, a memory leak, poorly written Java applications, not enough Perm Gen space defined, poor garbage collection settings, not enough physical memory being available on the machine where the JVM runs. Normally when a container runs out of memory it will be displayed in the server logfiles.

Types of out of memory errors that may be seen in the server logfiles:

“Caused By: java.lang.OutOfMemoryError: Java heap space”

This indicates that the maximum defined heap space in -Xmx has been exceeded. It could be that the maximum heap is too small, the system is overloaded, there is a memory leak, Eden Space (Young Gen) is not allocated at an ideal percentage of the maximum heap, or the garbage collection scheme is not optimal for the applications running in the JVM.

“Caused by: java.lang.OutOfMemoryError: PermGen space”

  • The PermSpace contains
  • Class information (not objects)
  • Constant strings and dynamic String.intern() objects
  • Hotspot compiled code
  • PermSpace is independent of Young/Old Space
  • It lives outside the region specified by Xms/Xmx

The PermSpace is normally not a target of tuning efforts, usually more space must be allocated in the -XX:PermSize/-XX:MaxPermSize parameters.

There are a few exceptions though:

  • Applications that do dynamic class loading and unloading might suffer from a too small PermSpace
  • A PermSpace can trigger unnecessary Full GC activity with the Concurrent-Mark-Sweep (CMS) collector
  • An oversized PermSpace unnecessarily increases memory footprint which in turn increases the likelihood to hit the 2 or 4 g process size limit

“java.lang.OutOfMemoryError: unable to create new native thread”

 This HotSpot JVM error is thrown when the internal JVM native code is unable to create a new Java thread. More precisely, it means that the JVM native code was unable to create a new “native” thread from the OS.

Possible solutions:

  • Set kernel parameter maxdsiz to a higher value
  • Reduce the current heap size
  • Check the kernel values: ulimit -a
  • If the NPROC soft limit is lower than the hard limit, increase it as needed: ulimit -u <new value>, check the Operating System documentation to make changes permanent at the OS configuration files
  • Need to reduce the JVM stack size and the OS stack size both, set the -xss in java options and set ulimit -s on the OS level

Diagnostic Steps 

—————————————————————————————————————————————-

  • What OS and version?
  • What JVM and version?
  • How much RAM available on the machine?
  • Is there available RAM to increase the heap size if  necessary?
    • On Linux, obtain meminfo output when the Java application is running:

  • What JVM options are being used?
    • On JBoss EAP, options are output to the beginning of the console log or boot.log.  Verify the desired heap settings are being properly found and applied.
  • Is  this a newly deployed application, or has it been in production for  some time and only recently this issue has appeared.
  • Are there any particular circumstances surrounding when the  OutOfMemoryError happens? For example, does it happen when specific  functionality is executed (e.g. a file upload, a batch job, etc.), after a certain amount of time, under high load, etc.
  • Get a heap dump (How do I create a Java heap dump?) when the issue happens and analyze it (How do I analyze a Java heap dump?) to see where the retention points are.