Skip to main content

startup fails due to partially written install configuration files from karaf bootstrap

Hajime
Updated

SYMPTOM:

After an OS restart the service to restart Nexus Repo is sporadically failing to start.  When this problem occurs the following is seen in the logs:

2021-03-03 23:37:22,644+0000 ERROR [FelixStartLevel]  *SYSTEM org.apache.karaf.deployer.features.FeatureDeploymentListener - Unable to update deployed features for bundle: org.apache.felix.framework - 5.6.12
java.lang.NullPointerException: null
 at org.apache.karaf.deployer.features.FeatureDeploymentListener.bundleChanged(FeatureDeploymentListener.java:247)
 at org.apache.karaf.deployer.features.FeatureDeploymentListener.init(FeatureDeploymentListener.java:95)
 at org.apache.karaf.deployer.features.osgi.Activator$DeploymentFinishedListener.deploymentEvent(Activator.java:86)
 at org.apache.karaf.features.internal.service.FeaturesServiceImpl.registerListener(FeaturesServiceImpl.java:295)
 at org.apache.karaf.deployer.features.osgi.Activator.doStart(Activator.java:53)
 at org.apache.karaf.util.tracker.BaseActivator.start(BaseActivator.java:92)
 at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697)
 at org.apache.felix.framework.Felix.activateBundle(Felix.java:2240)
 at org.apache.felix.framework.Felix.startBundle(Felix.java:2146)
 at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1373)
 at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
 at java.lang.Thread.run(Thread.java:748)

or:

2021-03-93 21:27:20,304-0400 WARN [FelixStartLevel] *SYSTEM org.ops4j.pax.url.mvn.internal.AetherBasedResolver - Error resolving artifact org.apache.karaf.features:standard:xml:features:4.3.9: [Could not transfer artifact org.apache.karaf.features:standard:xml:features:4.3.9 from/to central (https://repo1.maven.org/maven2/): transfer failed for https://repo1.maven.org/maven2/org/apache/karaf/features/standard/4.3.9/standard-4.3.9-features.xml]
java.io.IOException: Error resolving artifact org.apache.karaf.features:standard:xml:features:4.3.9: [Could not transfer artifact org.apache.karaf.features:standard:xml:features:4.3.9 from/to central (https://repo1.maven.org/maven2/): transfer failed for https://repo1.maven.org/maven2/org/apache/karaf/features/standard/4.3.9/standard-4.3.9-features.xml]
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:555)
at org.ops4j.pax.url.mvn.internal.Connection.getInputStream(Connection.java:123)
at java.net.URL.openStream(URL.java:1093)
at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:114)
at org.apache.karaf.features.internal.service.RepositoryImpl.<init>(RepositoryImpl.java:51)
at org.apache.karaf.features.internal.service.RepositoryCacheImpl.create(RepositoryCacheImpl.java:51)
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:386)
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:381)
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.addRepositories(BootFeaturesInstaller.java:136)
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:95)
at org.apache.karaf.features.internal.service.BootFeaturesInstaller.start(BootFeaturesInstaller.java:89)
at org.apache.karaf.features.internal.osgi.Activator.doStart(Activator.java:215)
at org.apache.karaf.util.tracker.BaseActivator.start(BaseActivator.java:92)
at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:698)
at org.apache.felix.framework.Felix.activateBundle(Felix.java:2402)
at org.apache.felix.framework.Felix.startBundle(Felix.java:2308)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1539)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:750)
Suppressed: shaded.org.eclipse.aether.transfer.ArtifactTransferException: ...

CAUSE:

This exception happens when at least one of karaf configuration files is truncated or corrupted.

RESOLUTION:

Upgrade to the version 3.57.1 or higher, as the ${installDirectory}/etc/karaf/config.properties has "felix.fileinstall.enableConfigSave = false" (NEXUS-27710), which should prevent this issue.

 

For older versions:

How to find which file is corrupted:

_VER="3.40.1-01";
_INSTALL_DIR="/opt/sonatype/nexus-${_VER}"

cd $(dirname ${_INSTALL_DIR%/});
curl -o "/tmp/nexus-unix.tar.gz" -L "https://download.sonatype.com/nexus/3/nexus-${_VER}-unix.tar.gz";
tar --diff -f "/tmp/nexus-unix.tar.gz" nexus-${_VER}/etc/karaf | grep -vE '(Uid|Gid|Mod time) differs'

Example commands for restoring all files under "karaf" directory:

cd $(dirname ${_INSTALL_DIR%/});
tar -xv -f "/tmp/nexus-unix.tar.gz" nexus-${_VER}/etc/karaf

After restoring the corrupted file, also change "felix.fileinstall.enableConfigSave" to "false"

sed -i.bak -E 's/felix.fileinstall.enableConfigSave *= *true/felix.fileinstall.enableConfigSave = false/' ${_INSTALL_DIR%/}/etc/karaf/config.properties

Related articles