Understanding Eclipse Jetty 9.4.8 Thread Allocation

Overview

The following article attempts to explain via sample log messages and code references the default connector thread allocation logic for Eclipse Jetty 9.4.8, which is used in Nexus Repository Manager 3.9.0 and greater.

See the Thread Starvation and Always two there are sections of

https://webtide.com/thread-starvation-with-eat-what-you-kill-2/

Examples in this document are based on using a 160 CPU host.

High-level Eclipse Jetty Documentation

https://www.eclipse.org/jetty/documentation/current/architecture.html https://www.eclipse.org/jetty/documentation/current/quickstart-config-what.html https://www.eclipse.org/jetty/documentation/current/high-load.html - suggests thread pool size be > 50 and < 500

Related Mailing List Posts

https://dev.eclipse.org/mhonarc/lists/jetty-users/msg04751.html

QueuedThreadPool

Jetty pools threads that are responsible for its work. The default pool is sized with a minimum capacity of 8 and a maximum capacity of 200.

The default implementation is the class org.eclipse.jetty.util.thread.QueuedThreadPool .

Thread Use Log Messages

QueuedThreadPool.toString()

return String.format("QueuedThreadPool@%s{%s,%d<=%d<=%d,i=%d,q=%d}", 
_name,
getState(),
getMinThreads(),
getThreads(),
getMaxThreads(),
getIdleThreads(),
(_jobs == null ? -1 : _jobs.size()));

Example:

QueuedThreadPool@qtp2027989430{STARTED,8<=8<=12,i=6,q=0}

ThreadPoolBudget

Jetty can be configured to print warn log messages at a configured threshhold which by default is the number of processors.

https://github.com/eclipse/jetty.project/blob/jetty-9.4.8.v20171121/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ThreadPoolBudget.java#L100

This sample log message was printed when 3 connectors were already present in the system, just before a 4th one was about to be added - note that it tells us there are only 32 threads available from the pool to allocate from the default 200 max threads in the pool.

2018-02-21 08:40:14,860+0100 WARN  [qtp175712895-451310]  I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - Low configured threads: (max=200 - required=168)=32 < warnAt=160 for QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}

AbstractConnector - Acceptor Threads

A connector requires acceptor threads from the thread pool:

acceptors=Math.max(1, Math.min(4,cores/8))

Detailed Acceptors Advise in Javadoc

Sample Log message when about to start a new connector on port 8086 on a 160 CPU host:

2018-02-21 08:40:14,860+0100 INFO  [qtp175712895-451310]  I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - ServerConnector@46644a4b{HTTP/1.1,[http/1.1]}{10.17.81.108:8086} requires 4 threads from QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}

SelectorManager - Selector Threads

A Connector requires selector threads and uses a SelectorManager instance.

For a sized thread pool(default), Jetty calculates the number of selectors needed using this logic:

int threads = ((ThreadPool.SizedThreadPool)executor).getMaxThreads();
int cpus = Runtime.getRuntime().availableProcessors();
return Math.max(1,Math.min(cpus/2,threads/16));

In this sample log message the SelectorManager reports it requires 12 threads from the thread pool for selector threads max(1, min(160/2=80,200/16=12.5) ) = 12:

2018-02-21 08:40:14,861+0100 INFO  [qtp175712895-451310]  I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - SelectorManager@ServerConnector@46644a4b{HTTP/1.1,[http/1.1]}{10.17.81.108:8086} requires 12 threads from QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}

ReservedThreadExecutor

New in Jetty 9.4.x as compared to Jetty 9.3.x, as explained in the webtide blog post, Jetty also tries to reserve a number of threads from a pool ( the same pool as all other Jetty threads by default ) to be used if Jetty is being heavily loaded with requests and the free threads in the pool are not enough to handle the load. This is handled by the ReservedThreadExecutor.

The default heuristic used to calculate the number of reserved threads needed is:

int threads = ((ThreadPool.SizedThreadPool)executor).getMaxThreads();
return Math.max(1, Math.min(cpus, threads / 8))

In this sample message on a 160 CPU host, the reserved thread executor requires 25 threads from the thread pool max(1, ,min(160,200/8=25))=25:

2018-02-21 08:40:14,861+0100 INFO  [qtp175712895-451310]  I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - ReservedThreadExecutor@5d8b5190{s=0/25,p=0}@SelectorManager@ServerConnector@46644a4b{HTTP/1.1,[http/1.1]}{10.17.81.108:8086} requires 25 threads from QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}

Starting a connector without sufficient pooled threads

In our example, we already have 3 connectors started on a 160 CPU host and a system under load.

When a connector starts, the selectors and reserved threads need to allocate first. In that case, Jetty needs 25 reserved threads + 12 selector threads = 37 threads to be available. We can ignore the 4 acceptor threads for now, as they will only be needed if the other threads can be allocated.

Our log messages indicate there are only 32 threads available in the pool.

Since 37 threads required for the new connector > 32 available in the pool, starting the connector fails ( 168+37=205 ) with this log message:

2018-02-21 08:40:14,862+0100 WARN [qtp175712895-451310] I312505 org.sonatype.nexus.bootstrap.jetty.ConnectorManager - Could not start connector: DockerConnectorConfiguration{repositoryName=deploy.milestones.docker, scheme=http, port=8086} java.lang.IllegalStateException: Insufficient configured threads: required=205 < max=200 for QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}
Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk