.

1. Option 3 - HA Deployment in AWS

1.1 ProvisioningFailed

kubectl describe pvc/nexus-data-nxrm-ha-aws-61-0-0-nxrm-statefulset-0 -n nexusrepo

How to know to describe the PVC?

1.1.1 Nexus Repository 3 resources status 

1.1.1.1 Pods

The Nexus Repository 3 HA AWS helm chart creates 3 pods by default. But there is only 1 pod, which is in Pending status.

There are no events after describing the pod if the AGE of the pod is younger than 10m

But if the AGE is older than 10m:

1.1.1.2 StatefulSet

1.1.1.3 PVC

1.1.2 Understand the error

The error means there is a permission issue of ebs.csi.aws.com when creating the required PV. Recall

• If you are using EKS version 1.23+, you must first install the AWS EBS CSI driver before running the current AWS Helm chart. We recommend using the EKS add-on option as described in AWS’s installation instructions.

1.1.3 How to solve the issue

AWS Management Console tab in https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html 

1.) Login to AWS Console -> EKS -> Check the EKS cluster has the required OICD

2.) Go to the Add-ons tab->Search EBS

2.1) Check if the role's name is correct; if not, don't modify it. Instead, delete the Add-on and readd the EBS CSI Driver Add-on with the correct role.

2.2) Click the role and check the role's permission

3.) Check the role's permission policy is AmazonEBSCSIDriverPolicy and is AWS Managed. This role shouldn't be modified unless there is a good reason and you know what you are doing. 

4.) Check the role's Trust relationships are correct. Any typo can cause an issue

An example:

1.2 FailedMount

1.2.1 Error 1

Error from describing the pod: 

or 

This error is clear. The Secret Store CSI Driver wasn't installed.

Step 6 - AWS Secrets Manager

Follow the AWS documentation for Secrets Store CSI Drivers to mount the license secret, which is stored in AWS Secrets Manager, as a volume in the pod running Nexus Repository.

Error from describing the pod: 

Besides installing the Secret Store CSI Driver, the Secrets Manager and Config Provider are also required:

Installing the AWS Provider

To install the Secrets Manager and Config Provider use the YAML file in the deployment directory:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

1.2.2 Error 2

how to solve it:

From the error message, check the role secrets-store-csi-service-account-role has the required policy to get the secrets values. 

1.2.3 Error 3

Error from describing the pod: 

how to solve it:

The issue is similar to 1.1. 

Please check the role defined in the below and its Trust relationships

https://github.com/sonatype/nxrm3-ha-repository/blob/main/nxrm-ha-helm/values.yaml#L25

If the below is defined in Trust relationships:

"oidc.eks.ca-central-1.amazonaws.com/id/2F203DF611EB4531E51B812FDAEC12FD:sub": "system:serviceaccount:nexusrepo:nexus-repository-deployment-sa"

Then the serviceaccount is the one defined in:

https://github.com/sonatype/nxrm3-ha-repository/blob/main/nxrm-ha-helm/values.yaml#L24

the value is in the format: system:serviceaccount:<serviceaccount-namespace>:<service-account>

1.2.4 Error 4

Error from describing the pod: 

check the log of the secrete store provider

kubectl logs -l app=csi-secrets-store-provider-aws --prefix -n kube-system

If there is JMES Path in the log:

 Then check if the secret key is the same as the JmesPath. Use branch 3.61.0-02 as an example:

For the secret:

https://github.com/sonatype/nxrm3-ha-repository/blob/61.0.2/nxrm-aws-ha-helm/templates/secret.yaml#L35

It should have a key-value pair with the key of the same as

https://github.com/sonatype/nxrm3-ha-repository/blob/61.0.2/nxrm-aws-ha-helm/templates/secret.yaml#L37

Which by default is admin_nxrm_password

The same for the secret:https://github.com/sonatype/nxrm3-ha-repository/blob/61.0.2/nxrm-aws-ha-helm/templates/secret.yaml#L27

the secret should have 3 key-value pairs whose keys are: username, password, and host

https://github.com/sonatype/nxrm3-ha-repository/blob/61.0.2/nxrm-aws-ha-helm/templates/secret.yaml#L28-L34

1.3 Can't find secret

Error from describing the pods: 

The secret should be synced by Secrets Store CSI Drivers. To see more information about the error, we need to check the logs of Secrets Store CSI Drivers. DaemonSet manages the CSI Driver, creating 1 pod(csi-secrets-store-secrets-store-csi-driver-xxxxx) in each node. If there are only 3 nodes, then checking the logs of each pod is endurable. But if there are many, then it's pretty troublesome to go through each pod of the CSI Driver. Another way is to check all the CSI driver pods by label(the limitation is it only shows 10 latest log entries for each pod)

kubectl logs -l app=secrets-store-csi-driver --prefix -n kube-system

1.3.1 How to solve the issue

Check the value of the syncSecret.enable.

helm get values csi-secrets-store -n kube-system 

Output:

USER-SUPPLIED VALUES:
syncSecret:
     enabled: false

helm upgrade csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true -n kube-system