Nexus Repository 2 Thread Dump and Heap Dump Guide

 

Visit my.sonatype.com for documentation on Nexus Repository version 2.

REST thread dump - Nexus Repository 2.5+
curl -u admin:admin123 http://localhost:8081/nexus/internal/threads > nexus.tdump

Print the Nexus Repository 2 PID
ps -e -o pid,command | /bin/egrep -o '^ *[0-9]{1,5} *java.*wrapper' | /bin/egrep -o '^ *[0-9]{1,5}'
This works if there is only one Nexus Repository 2 instance on your host.

Quick thread dump
kill -QUIT <nexus_pid> or kill -3 <nexus_pid>
This does not stop(kill) the Nexus Repository 2 process - merely tells the JVM to take a thread dump. Dump is in wrapper.log - simply send the zipped wrapper logs, sending multiple log files if the log files have rolled over during the thread dump.

Thread dump with locking details
jstack -l <nexus_pid>
The "-l" switch will provide detailed information about the locks that are held. Use same java version and user of that running nexus ( thread dump in wrapper.log )


Introduction

When Nexus Repository 2 is not performing well, the most common step to troubleshoot the issue besides examining DEBUG logs, is to gather thread dumps during the performance issue.

Preparation

The Nexus Repository 2 bundle uses the Java Service Wrapper (JSW) to control the Nexus Repository 2 application process. In practice, some default settings in Nexus Repository prior to 2.5 made getting a thread dump from the command line in some circumstances more difficult. To make it easier to get a thread dump from Nexus Repository, we suggest the following changes to the JSW configuration in versions prior to Nexus Repository 2.5.

Disable wrapper restart after ping timeout to allow time for heap/thread dump

Set wrapper.ping.timeout=0 in "<nexus_root>/bin/jsw/conf/wrapper.conf"

Note: This is the default starting in Nexus Repository 2.5 - see NEXUS-5678

Disable wrapper startup timeout to allow time for Nexus Repository to start up, jetty port accessible

Set wrapper.startup.timeout=0 in "<nexus_root>/bin/jsw/conf/wrapper.conf"

Note: This is the default starting in Nexus Repository 2.5 - see NEXUS-5678

Getting a Thread Dump in Nexus 2.5+

Nexus Repository 2 now exposes a simple REST call to get a minimal thread dump. Send a request to the following URL as a Nexus Repository Administrator or a user assigned the "Metrics Endpoints" role or permission.

http://localhost:8081/nexus/internal/threads

Note: The downside of this method is that it does not print heap information as a regular thread dump or jstack -l would.

Getting a Complete Thread Dump in All Nexus Repository 2 Versions

Unix/Linux

Finding the Nexus Repository 2 Application Process ID (PID)

In order to trigger a thread dump from a terminal, you need to know the Nexus Repository 2 application PID. When the standard bundle is launched, two processes are started - initially the JSW wrapper process - then the actual Nexus Repository 2 Application process.

The Nexus Repository 2 control script has a 'status' cmd. The ./bin/nexus status command will print a process id. For example:

Nexus Professional is running (6893).

However, this PID will be the Java Service Wrapper process, not the Nexus Repository 2 application process. You cannot get a thread dump from the wrapper PID. Instead, follow steps 1 and 2 below.


Trigger a Thread Dump From the Command Line

Note: Please ensure you are running these commands as the user running the Nexus Repository 2 process.

Single Command

ps -e -o pid,command | /bin/egrep -o '^ *[0-9]{1,5} *java.*wrapper' | /bin/egrep -o '^ *[0-9]{1,5}' | xargs kill -3

Alternative

1) ./bin/nexus status

If Nexus Repository 2 is running, a PID of the JSW  wrapper process is in brackets. Note this number and proceed to step 2.

2) ps -e -o pid,ppid,command | grep wrapper.pid=<wrapper_pid> | grep nexus

 Replace <wrapper_pid> with the PID from step 1. The number that starts the resulting output should be the Nexus Application PID. Note this number and proceed to step 3

3) kill -QUIT <nexus_pid> or kill -3 <nexus_pid>

The complete thread dump should get printed in the <nexus_app>/logs/wrapper.log file.

Note: In the case of a large number of threads, the wrapper.log may roll over and be spread across more than one log file. You can adjust the size of the wrapper log file to prevent this if getting a complete thread dump in this manner is proving difficult.

Windows

Getting a thread dump from a Java process in Windows is often more difficult that under *nix based systems. 

Getting a Heap Dump

In very rare cases we may suggest triggering a heap dump from your Nexus Repository 2 instance. This is rare because %99 of the time, a thread dump + detailed logs will provide enough context for us to troubleshoot your issue. Triggering a heap dump can itself cause instability problems in a JVM if resources are being strained. It also can consume a lot of disk space.

However, in the case of your Nexus Repository 2 has thrown an OutOfMemoryException, a heap dump may prove extremely useful to determine what was consuming all of the memory.

Manual Heap Dump
jmap -dump:format=b,file=heapdump.bin <nexus_pid>

Automatic Heap Dump on Oracle JVM OOM
Edit $NEXUS_HOME/bin/jsw/conf/wrapper.conf. Add a line wrapper.java.additional.4=-XX:+HeapDumpOnOutOfMemoryError, where 4 is the next available unused number for java additional properties.

Have more questions? Submit a request

0 Comments

Article is closed for comments.