Skip to main content

How to resize Persistent Volumes in a Nexus Repository HA Kubernetes Deployment

Updated

Overview

This article provides a guide on resizing Persistent Volumes in a Nexus High Availability (HA) Kubernetes deployment. While Kubernetes natively supports resizing Persistent Volumes, as detailed in the official documentation Resizing Persistent Volumes using Kubernetes (please read it before proceeding with the steps below), this article includes additional steps specific to a Nexus HA deployment to ensure a smooth and effective resizing process.

Steps

1. Estimating the Required Size

Before resizing a Persistent Volume, it's crucial to estimate the required size based on current usage and anticipated future growth. In a Nexus HA deployment with multiple pods, cloud-based blob storage (such as AWS S3 or Azure Blob Storage) or shared file storage are used to store components, with PostgreSQL handling metadata. In this scenario, the only data occupying Persistent Volumes are the logs.

By default, logs are stored in the /nexus-data/log directory. Each pod has its Persistent Volume (PV) and corresponding log directory. Logs are rotated daily and compressed, though task logs are not rotated. By default, Nexus Repository 3 retains 90 days' old logs with current log files(task logs, request.log, audit.log, and nexus.log).

There is no standardized method for estimating log size, as it can vary significantly depending on workloads, logger settings, and the tasks being executed. However, you can use the following approach to gauge an estimate.

  • Bash into a Pod and check the current log size (CURRENT_LOG_SIZE) using the following command: 
    du -sh /nexus-data/log

    Assume CURRENT_LOG_SIZE=1024MB.

  • Identify the oldest and latest archive log files (e.g., nexus-<date>.log.gz, request-<date>.log.gz, or audit-<date>.log.gz) in the log directory /nexus-data/log
  • Calculate the number of days between the oldest and latest log files. For example, if the oldest file is nexus-2024-08-23.log.gz and the latest is nexus-2024-09-01.log.gz, then DAYS=10, the average daily log size: 
    AVG_DAILY_LOG_SIZE = CURRENT_LOG_SIZE / DAYS = 1024MB / 10 days = 102.4MB/day
  • Estimate the required size for the new Persistent Volume: 
    ESTIMATED_SIZE = AVG_DAILY_LOG_SIZE * 100 days * 1.5 = 102.4MB/day * 100 days * 1.5 = 15GB

    Note:

    • 100 days are used to account for the 90 days' old logs plus an additional 10 days for current log files(task logs, request.log, audit.log, and nexus.log); we assume the total size of all current log files is 10 times larger than the average size.
    • The factor 1.5 accounts for any unexpected growth or spikes in log size. Adjust this factor based on your environment.
    • Even with the above estimate, it can't guarantee you won't have a "No Space" issue.

2. Resizing Persistent Volumes

2.1 Check if the value of the allowVolumeExpansion in the storage class is True

  • kubectl -n nexusrepo get pvc GET-PVC.png
  • kubectl -n nexusrepo describe sc managed-csi
    DescribeManagedCSISC.png
    Note:
    If the PVCs use one of the storage classes provided by AKS or EKS, the allowVolumeExpansion is True by default. If the value of the allowVolumeExpansion in the storage class is not True, you need to update the storage class to allow volume expansion.
    • kubectl -n nexusrepo edit sc nexus-storage add line allowVolumeExpansion: true in the storage class definition, then save and exit.
      SetAllowVolumeExpansionToTrue.png

2.2 Edit each pod's PVC to resize the Volume

  • kubectl -n nexusrepo edit pvc nexus-repository-manager-0 
  • Change the spec.resources.requests.storage value to the new size (e.g., 15Gi)
    ChangePVCSize.png
  • Save and exit
  • Describe the PVC to verify resizing succeeded: 
    kubectl -n nexusrepo describe pvc nexus-repository-manager-0 
    DescribePVC.png
  • Repeat the above steps for each PVC in the Nexus HA deployment
  • Check the capacity of the PVCs and PVs to ensure they have been resized: 
     
    kubectl -n nexusrepo get pvc
kubectl get pv
    GetNewPVC.pngGetNewPV.png

2.3 Update the Nexus HA Helm chart with the latest changes

  • (only for Nexus HA helm chart version 68.0.0 and newer). If the Nexus HA helm chart created the storage class, add the line allowVolumeExpansion: true to the storageClass section in the values.yaml file. 
      storageClass:
    allowVolumeExpansion: true

    To verify if the Nexus HA helm chart created the storage class:

    • Describe the storage class: kubectl -n nexusrepo describe sc nexus-storage check if it has an annotation meta.helm.sh/release-name=<your-helm-release-name>
    • Check the current release's values.yaml file:
    storageClass:
      # true and the below name matches the storage class you used for Nexus HA deployment    
  enabled: true 
  name: nexus-storage
  • Upgrade the pvc.storage to the new value in the values.yaml file: 
    pvc:
  storage: 15Gi
    The below steps will cause downtime. You don't have to do it right away. You can wait until you'd like to upgrade to a newer Nexus Repository version. You need to scale the replicas to 0 and delete the current statefulset, then upgrade. Otherwise, you may have the below error: 
    Error: UPGRADE FAILED: cannot patch "nexus371-nxrm-ha" with kind StatefulSet: 
    StatefulSet.apps "nexus371-nxrm-ha" is invalid: spec: Forbidden: updates to 
    statefulset spec for fields other than 'replicas', 'ordinals', 'template', 
    'up dateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' 
    are forbidden
  • Scale the StatefulSet to 0 replicas: kubectl -n nexusrepo scale statefulset nexus371-nxrm-ha --replicas=0
  • Wait until all pods are terminated, then delete the statefulset: kubectl -n nexusrepo delete statefulset nexus371-nxrm-ha
  • Upgrade the Nexus HA Helm chart with the values.yaml file after making the changes: helm upgrade nexus371 -f values.yaml

Related articles