.
Problem
If Nexus Repository or Sonatype IQ Server are run on a RedHat 8 host system using OpenJDK 8, these Sonatype server products may not start, read product license files, or perform other cryptographic operations successfully.
One can check the java system properties of our running Nexus Repository product for proof that these conditions may be met - for example:
"java.vendor" : "Red Hat, Inc.",
"java.vendor.url" : "https://www.redhat.com/",
"java.vendor.url.bug" : "https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-1.8.0-openjdk",
"java.version" : "1.8.0_322",
"java.vm.info" : "mixed mode",
"java.vm.name" : "OpenJDK 64-Bit Server VM",
"java.vm.specification.name" : "Java Virtual Machine Specification",
"java.vm.specification.vendor" : "Oracle Corporation",
"java.vm.specification.version" : "1.8",
"java.vm.vendor" : "Red Hat, Inc.",
"java.vm.version" : "25.322-b06",
Nexus Repository Symptoms
An example of logging from nexus.log can look like the following:
2021-06-28 23:01:33,974+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.node.LocalNodeAccess - Failed transition: NEW -> STARTED
org.sonatype.nexus.ssl.KeystoreException: Unable to retrieve key manager in keystore 'private.ks' for alias 'identity'
at org.sonatype.nexus.ssl.internal.geronimo.FileKeystoreInstance.getKeyManager(FileKeystoreInstance.java:261)
at org.sonatype.nexus.ssl.KeyStoreManagerImpl.getKeyManagers(KeyStoreManagerImpl.java:249)
at org.sonatype.nexus.ssl.KeyStoreManagerImpl.generateAndStoreKeyPair(KeyStoreManagerImpl.java:376)
at org.sonatype.nexus.internal.node.LocalNodeAccess.doStart(LocalNodeAccess.java:73)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
at org.sonatype.nexus.internal.node.NodeAccessBooter.start(NodeAccessBooter.java:48)
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.KeyStoreException: FIPS mode: KeyStore must be from provider SunPKCS11-NSS-FIPS
at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:67)
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256)
at org.sonatype.nexus.ssl.internal.geronimo.FileKeystoreInstance.getKeyManager(FileKeystoreInstance.java:256)
...
Another possible cause example that might be related to FIPS mode:
2024-01-16 09:57:53,441-0700 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.node.orient.OrientLocalNodeAccess - Failed transition: NEW -> STARTED
org.sonatype.nexus.ssl.KeystoreException: Unable to retrieve key manager in keystore ‘private.ks’ for alias ‘identity’
at org.sonatype.nexus.ssl.internal.geronimo.FileKeystoreInstance.getKeyManager(FileKeystoreInstance.java:262)
at org.sonatype.nexus.ssl.KeyStoreManagerImpl.getKeyManagers(KeyStoreManagerImpl.java:249)
at org.sonatype.nexus.ssl.KeyStoreManagerImpl.generateAndStoreKeyPair(KeyStoreManagerImpl.java:376)
at org.sonatype.nexus.internal.node.orient.OrientLocalNodeAccess.doStart(OrientLocalNodeAccess.java:76)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:206)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
at org.sonatype.nexus.internal.node.NodeAccessBooter.start(NodeAccessBooter.java:52)
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.security.KeyStoreException: Cannot get key bytes, not PKCS#8 encoded
at sun.security.provider.KeyProtector.protect(KeyProtector.java:161)
at sun.security.provider.JavaKeyStore.engineSetKeyEntry(JavaKeyStore.java:273)
at sun.security.provider.JavaKeyStore$JKS.engineSetKeyEntry(JavaKeyStore.java:57)
at sun.security.provider.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:117)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineSetKeyEntry(JavaKeyStore.java:71)
at java.security.KeyStore.setKeyEntry(KeyStore.java:1140)
at org.sonatype.nexus.ssl.internal.geronimo.FileKeystoreInstance.getKeyManager(FileKeystoreInstance.java:255)
Sonatype IQ Server Symptoms
When the product license is read, the following messages may appear in the clm-server.log, even though the license used/installed is valid:
com.sonatype.insight.brain.product.license.LicenseAwareContainerDynamicFeature$Filter - No valid product license installed.
com.sonatype.insight.brain.product.license.InvalidLicenseException: No valid product license installed.
Diagnosis
Redhat has enabled FIPS (Federal Information Processing Standard) by default - this feature is currently not compatible with Nexus Repository products.
- Official Reference Document:
https://access.redhat.com/documentation/en-us/openjdk/8/pdf/configuring_openjdk_8_on_rhel_with_fips/OpenJDK-8-Configuring_OpenJDK_8_on_RHEL_with_FIPS-en-US.pdf - Copy of the above document downloaded from that link on 2022-09-14 and attached to this article
- RedHat support resource page: https://access.redhat.com/solutions/5696401
Sonatype Tracking Issues
There is a product idea that authorized support customers can track and vote on for FIPs support.
https://ideas.sonatype.com/ideas/IDEAS-I-1610
Workarounds
There are a couple of ways to get Nexus Repository server products to start-up and load license files correctly.
1. (Preferred) Pass the following argument to the java command used to start Nexus Repository or the Sonatype IQ Server
-Dcom.redhat.fips=false
Consult the documentation of Sonatype IQ Server and Nexus Repository for distribution-specific ways to set this java system property.
["When set to false, disables the FIPS alignment while still applying the global crypto-policies.
If any of the previous properties is set to disable the crypto-policies alignment, this property
has no effect. In other words, crypto-policies is a prerequisite for FIPS alignment"]
2. (Less ideal) Pass the following argument to the java command used to start Nexus Repository or the Sonatype IQ Server
-Djava.security.disableSystemPropertiesFile=true
Consult the documentation of Nexus IQ Server and Nexus Repository for distribution specific ways to set this java system property.
["When set to true, both the global FIPS and the crypto-policies alignment are disabled;
generating the same effect than a security.useSystemPropertiesFile=false security
property. If both properties are set to different behaviors,
java.security.disableSystemPropertiesFile overrides."]
3. (Not-preferred) Edit the JDK/JRE global ./lib/security/java.security file and set security.useSystemPropertiesFile to 'false'. This affects ALL java applications using that version of Java on the same host.
["When set to false, both the global FIPS and the crypto-policies alignment are disabled. By
default, it is set to true"]