<TABLE OF CONTENTS>
Nexus Repository 2.15.0 now includes Eclipse Jetty 9. Previous releases of Nexus Repository 2 used some version of Eclipse Jetty 8.
This article discusses this change and the potential impacts.
Why was this change made?
Eclipse Jetty 8 has proven to be a reliable web container for Repository 2. However it has been end-of-lifed by the project maintainers for some time. It was decided continuing its further use was a maintenance risk we needed to address.
Eclipse Jetty 9 has been in use in Repository 3 for years and it has proven a reliable and stable platform. It is also still supported by the Jetty project maintainers.
To review all Eclipse Jetty 9 internal changes in detail, view the Jetty project changelog.
What are the Jetty related impacts when upgrading to Repository 2.15.0?
Here we summarize the major Jetty related considerations when upgrading.
Update to Latest Java 8 Strongly Recommended
Eclipse Jetty maintainers and Sonatype recommend using the latest Java 8 version available with our products. This helps avoid bugs and provides the strongest security and default settings. Take this opportunity to upgrade your version of Java 8 while upgrading to the new version of Repository 2.
Jetty Configuration File and Format Changes
The Jetty configuration files have changed and the new file formats must be used. Configuration files from previous versions are not drop-in compatible. The same capabilities are still available with slight differences in format and file names.
Jetty XML configuration files remain located under
<application-directory>/conf/ and are prefixed with jetty as before. The jetty-http-redirect-to-https.xml file has been removed.
You must start with the new configuration files and make edits directly to those. Any automation that copies or retain use of old Jetty specific configuration files should be reviewed. The upgrade procedures for Repository 2 have consistently recommended for years to edit new files on upgrade and not re-use old config files. Preparing these files can be done in advance of upgrading a production system, without requiring production downtime.
Jetty Thread Pool Size
Repository 2 releases used a default Jetty max thread pool size of 254 threads. The internal thread pool handling algorithms of Jetty 9 have changed from Jetty 8. The default max thread pool size shipped with Repository 2 has been changed to 400 to match the defaults used by Repository 3 which also uses Jetty 9.
Sonatype recommends trying the new default of max 400 threads in the jetty pool. If required the Jetty thread pool size of 400 can be adjusted by adding a property to conf/nexus.properties. Example reducing the max size to 300:
jetty.threadpool.maxThreads=300
Adjusting Jetty Configuration Files Used
Jetty configuration files used are still specified inside the wrapper.conf file using the wrapper.app.parameter values. The default list has changed from:
wrapper.app.parameter.1=./conf/jetty.xml wrapper.app.parameter.2=./conf/jetty-requestlog.xml
to:
wrapper.app.parameter.1=./conf/jetty.xml wrapper.app.parameter.2=./conf/jetty-requestlog.xml wrapper.app.parameter.3=./conf/jetty-http.xml
The reason for this change was make it easier to disable the plain HTTP connector without editing the contents of a jetty configuration file.
If you start Repository 2.15.0 and cannot connect to its plain HTTP port, then verify your wrapper.conf file is referencing the new jetty-http.xml file.
The instructions for adjusting the referenced Jetty Configuration files remain the same, as long as you update and test your own custom configuration to be compatible with Jetty 9.
Reverse Proxy X-Forwarded Headers
The handling of inbound (X-)Forwarded HTTP header algorithms have changed in Eclipse Jetty 9. Customers using these headers with a reverse proxy in front of Repository 2 have a very small risk of some differences in behaviour on upgrade especially if you have configured headers contrary to our general recommendations.
Eclipse Jetty 9 maintainers now recommend using a RFC 7239 style Forwarded header if possible, however the old style X-Forwarded are still supported. Additional background in issue NEXUS-25095.
HTTPS Connector Keystore Settings
The file jetty-https.xml is still present for enabling the inbound HTTPS connector. The same properties ( ie. application-port-ssl ) and enablement method can be used.
You may notice the default keystore passwords are now plain text literal "password" instead of the obfuscated versions like "OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v". This is not a security issue, the previous values were obfuscated using a well known algorithm. Changing to plain text values was justified by the related issue addressed in Repository 3.
Redirection of HTTP to HTTPS Discouraged
The file jetty-http-redirect-to-https.xml has been removed to discourage its use. If you still want to perform HTTP 303 redirection to your secure connector from the insecure connector, we have an article describing the risks and how to enable that feature again.
HSTS Enabled By Default
HTTP Strict Transport Security (HSTS) is an industry standard used to tell a web browser to access sites over HTTPS. The HTTPS connector shipped with Repository 2 now has this feature enabled by default as Repository 3 already does. We have an article discussing configuration options.
Disabling HTTP Connector Simplified
When a secure HTTPS connector is configured, then a common requirement is to disable the plain HTTP connector. The process to do this is now simpler.
Instead of commenting out the HTTP connector inside of jetty.xml, all one needs to do is adjust the wrapper.app.parameter properties in wrapper.conf to not include a reference to ./conf/jetty-http.xml.
Default TLS Connector Enabled Ciphers More Secure
The default allowed ciphers for inbound Jetty based HTTPS connectors were including ciphers that were less secure than what the industry currently recommends. The new defaults represent industry accepted best practices as implemented by Eclipse Jetty 9 project itself. In rare situations you may need to customize the allowed ciphers to be more lax to support older build clients.
Default TLS Connector Allowed Protocols More Secure
Only TLSv12 is allowed by default for inbound HTTPS connections - other protocols are disabled. TLSv13 can be enabled as well if you are running the latest JRE 8. If you require supporting a different set of TLS protocols then refer to our article on how to make adjustments.
TLS Connector Protection Protection Added for Logjam Attacks
Previously one would need to explicitly enable protection against a type of attack called "Logjam" on weak inbound SSL Connections. Repository 2 now ships with protection against this attack enabled by default.
Adjusting Connector Idle Timeout
The default inbound connector connection idle timeout in Repository 2 remains configured as 30 seconds ( 30000ms ) and matches the default connector idle timeout of Jetty 9.4. Sonatype recommends leaving this default.
If the connection idle timeout is exceeded frequently, this could indicate a networking problem or resource constraint that needs investigation. In rare cases customers have had to increase the connection idle timeout by editing jetty xml configuration files.
Administrators can now customize the idle timeout for the inbound HTTP or HTTPS connectors by setting a value in milliseconds for the property jetty.http.idleTimeout or jetty.https.idleTimeout respectively. One only need to add these properties to conf/nexus.properties - no editing of a jetty xml configuration file is required. Example:
jetty.http.idleTimeout=30000
jetty.https.idleTimeout=30000
Using Java Management Extensions (JMX)
Repository 2 includes a jetty-jmx.xml file which when enabled helps to expose MBean instances to an external monitoring tool such as JConsole, VisualVM and Java Mission Control(MC).
Even when that file was not explicitly used, previous versions of Repository 2 exposed a limited set of Jetty related MBeans for local monitoring, such as org.eclipse.jetty.util.thread.QueuedThreadPool .
Version 2.15.0 still includes the jetty-jmx.xml file and it is configured in the same way as before. When enabled, it starts an RMIRegistry that external monitoring tools can connect to and read MBean metadata and possibly trigger operations.
The full set of MBeans exposed has been expanded to include Repository metrics ( nexus.metrics ) and more Jetty and JVM related internals. Customers who have monitoring against specific MBeans via JMX should review the newly exposed MBeans as these changes could affect your monitoring algorithms.
Custom Jetty Handlers and Servlet Filters
An example handler modification Sonatype has documented is filtering inbound access by IP address. For Jetty 9, the configuration will be slightly different.
0 Comments