.
The following article attempts to explain via sample log messages and code references from Jetty 9.4.8, the default connector thread allocation logic for Eclipse Jetty 9.4, which is used starting in Nexus Repository 3.9.0 and Nexus Repository 2.15.0.
Related Resources
Eclipse Jetty Blog Posts
See the Thread Starvation and Always two there are sections of https://webtide.com/thread-starvation-with-eat-what-you-kill-2/.
Eclipse Jetty Documentation
https://www.eclipse.org/jetty/documentation/jetty-9/index.html#basic-architecture
https://www.eclipse.org/jetty/documentation/jetty-9/index.html#quick-start-configure
https://www.eclipse.org/jetty/documentation/jetty-9/index.html#high-load
https://www.eclipse.org/jetty/documentation/jetty-9/index.html#limit-load
Eclipse Jetty Mailing List Posts
https://dev.eclipse.org/mhonarc/lists/jetty-users/msg04751.html
Analysis
Examples in this document are based on using a 160-CPU host.
QueuedThreadPool
Jetty pools threads that are responsible for its work. The default pool is sized with a minimum capacity of 8 and a maximum capacity of 200.
The default implementation is the class org.eclipse.jetty.util.thread.QueuedThreadPool.
Starting in Nexus Repository 3.13.0, the default Jetty configuration sets the maximum threads to 400.
Thread Use Log Messages
return String.format("QueuedThreadPool@%s{%s,%d<=%d<=%d,i=%d,q=%d}",
_name,
getState(),
getMinThreads(),
getThreads(),
getMaxThreads(),
getIdleThreads(),
(_jobs == null ? -1 : _jobs.size()));
Example:
QueuedThreadPool@qtp2027989430{STARTED,8<=8<=12,i=6,q=0}
ThreadPoolBudget
Jetty can be configured to print warn log messages at a configured threshold which by default is the number of processors.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.8.v20171121/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ThreadPoolBudget.java#L100
This sample log message was printed when 3 connectors were already present in the system, just before a 4th one was about to be added - note that it tells us there are only 32 threads available from the pool to allocate from the default 200 max threads in the pool.
2018-02-21 08:40:14,860+0100 WARN [qtp175712895-451310] I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - Low configured threads: (max=200 - required=168)=32 < warnAt=160 for QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}
AbstractConnector - Acceptor Threads
A connector requires acceptor threads from the thread pool:
acceptors=Math.max(1, Math.min(4,cores/8))
Detailed Acceptors Advise in Javadoc
Sample Log message when about to start a new connector on port 8086 on a 160 CPU host:
2018-02-21 08:40:14,860+0100 INFO [qtp175712895-451310] I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - ServerConnector@46644a4b{HTTP/1.1,[http/1.1]}{10.17.81.108:8086} requires 4 threads from QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}
SelectorManager - Selector Threads
A Connector requires selector threads and uses a SelectorManager instance.
For a sized thread pool(default), Jetty calculates the number of selectors needed using this logic:
int threads = ((ThreadPool.SizedThreadPool)executor).getMaxThreads(); int cpus = Runtime.getRuntime().availableProcessors(); return Math.max(1,Math.min(cpus/2,threads/16));
In this sample log message, the SelectorManager reports it requires 12 threads from the thread pool for selector threads max(1, min(160/2=80,200/16=12.5) ) = 12:
2018-02-21 08:40:14,861+0100 INFO [qtp175712895-451310] I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - SelectorManager@ServerConnector@46644a4b{HTTP/1.1,[http/1.1]}{10.17.81.108:8086} requires 12 threads from QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}
ReservedThreadExecutor
New in Jetty 9.4.x as compared to Jetty 9.3.x, as explained in the webtide blog post, Jetty also tries to reserve a number of threads from a pool ( the same pool as all other Jetty threads by default ) to be used if Jetty is being heavily loaded with requests and the free threads in the pool are not enough to handle the load. This is handled by the ReservedThreadExecutor.
The default heuristic used to calculate the number of reserved threads needed is:
int threads = ((ThreadPool.SizedThreadPool)executor).getMaxThreads(); return Math.max(1, Math.min(cpus, threads / 8))
In this sample message on a 160 CPU host, the reserved thread executor requires 25 threads from the thread pool max(1, ,min(160,200/8=25))=25:
2018-02-21 08:40:14,861+0100 INFO [qtp175712895-451310] I312505 org.eclipse.jetty.util.thread.ThreadPoolBudget - ReservedThreadExecutor@5d8b5190{s=0/25,p=0}@SelectorManager@ServerConnector@46644a4b{HTTP/1.1,[http/1.1]}{10.17.81.108:8086} requires 25 threads from QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}
Scenario: Starting a connector without sufficient pooled threads
In our example, we already have 3 connectors started on a 160 CPU host and a system under load.
When a connector starts, the selectors and reserved threads need to allocate first. In that case, Jetty needs 25 reserved threads + 12 selector threads = 37 threads to be available. We can ignore the 4 acceptor threads for now, as they will only be needed if the other threads can be allocated.
Our log messages indicate there are only 32 threads available in the pool.
Since 37 threads required for the new connector > 32 available in the pool, starting the connector fails ( 168+37=205 ) with this log message:
2018-02-21 08:40:14,862+0100 WARN [qtp175712895-451310] org.sonatype.nexus.bootstrap.jetty.ConnectorManager - Could not start connector: DockerConnectorConfiguration{repositoryName=deploy.milestones.docker, scheme=http, port=8086} java.lang.IllegalStateException: Insufficient configured threads: required=205 < max=200 for QueuedThreadPool@qtp175712895{STARTED,8<=200<=200,i=98,q=0}