Skip to main content

Nexus HA deployment on EKS fails with CrashLoopBackOff

Updated


OVERVIEW:

Nexus HA deployments on EKS cluster using Sonatype's helm chart requires per node local storage for work directory contents. This storage can be of EBS / NFS or Azure storage type.

 

ISSUE:

Nexus HA helm chart uses init container to change the ownerships of this work directory to nexus user ID 200, before starting the nexus container. When using shared storages, like NFS, this command may fail if NFS mount user id is different. This causes the init container to result in crash loop.

 

$ kubectl logs -f <helm-rel-name>-nxrm-ha-0 
│ chown: /nexus-data/etc/logback: Operation not permitted │
│ chown: /nexus-data/etc/logback: Operation not permitted │
│ chown: /nexus-data/etc: Operation not permitted │
│ chown: /nexus-data/etc: Operation not permitted │
│ chown: /nexus-data/log/tasks/allTasks.log: Operation not permitted │
│ chown: /nexus-data/log/tasks: Operation not permitted │
│ chown: /nexus-data/log/tasks: Operation not permitted │
│ chown: /nexus-data/log/request.log: Operation not permitted │
│ chown: /nexus-data/log/audit/audit.log: Operation not permitted │
│ chown: /nexus-data/log/audit: Operation not permitted │
│ chown: /nexus-data/log/audit: Operation not permitted │
│ chown: /nexus-data/log: Operation not permitted │
│ chown: /nexus-data/log: Operation not permitted │
│ chown: /nexus-data: Operation not permitted │
│ chown: /nexus-data: Operation not permitted │
│ Stream closed EOF for nexusrepo/nxha1-nxrm-ha-0 (chown-nexusdata-owner-to-nexus-and-init-log-dir)

 

CAUSE:

We see above permission error, generally when using shared file system and underlying directory do not have correct user id mapped. When using EFS storage, we can check the Access Points to confirm the POSIX user permission as in below screenshot.

 

 

 

SOLUTION:

Change the underlying storage directory permission to nexus user id 200. 

For EFS storage, we can use custom Storage Class, with correct gidRangeStart for nexus HA and use the same in our helm chart.

cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nx-efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: ${EFSID}
  directoryPerms: "700"
  gidRangeStart: "200"
  gidRangeEnd: "201"
  basePath: "/"

 

helm install nxha1 -n nexusrepo -f values.yaml \
--set storageClass.name=nx-ef-sc \
sonatype/nxrm-ha

Related articles