Skip to main content

How to Determine Sonatype Lifecycle Continuous Monitoring Activity and Performance

Updated

.

Continuous Monitoring, while not associated with CI/CD build tools, allows you to monitor Lifecycle Applications for newly discovered vulnerabilities, using the latest scanned components of an application in a stage, and using the latest policies.

https://help.sonatype.com/en/continuous-monitoring.html

https://help.sonatype.com/en/continuous-monitoring-best-practices.html

Continuous Monitoring Execution Review

  • Continuous Monitoring job runs daily at 12 AM (default configuration).
  • It checks all applications for existing scan results at configured stageas (e.g., "release") and then rescans matching applications one-by-one.
  • In an HA environment, the continuous monitoring job runs on only one node/server, processing all records. There is no way to configure which specific server node it runs on. The specific node may vary each time the task runs.
  • To preserve resources and prevent Sonatype services throttling and abuse, the job is single threaded by default

Is Continuous Monitoring Too Slow?

A reasonable expectation is to complete continuous monitoring for all apps and stages in less than 24 hrs.

If your continuous monitoring takes less than 24 hrs, there is no need to make adjustments.

If your continuous monitoring takes more than 24 hrs, it is not necessarily something that needs adjustment. Verify that you are following our best practices and make adjustments to stages monitored and policies scanned to reduce total execution time: https://help.sonatype.com/en/continuous-monitoring-best-practices.html

If your continuous monitoring takes more than 36 hrs, and you have made all other optimizations, consider adjusting the thread pool as outlined in this article.

Determining Continuous Monitoring Activity and Performance

You can examine IQ Server application log ( clm-server.log  ) files for related log messages:

  1. Identify which server HA node running Continuous Monitoring job: 
    Look for log entries such as:
    com.sonatype.insight.brain.policy.evaluator.PolicyMonitor - Starting policy monitoring
    com.sonatype.insight.brain.policy.evaluator.PolicyMonitor - Starting policy monitoring of applications
  2. Determine the number of application threads that will process all continuous monitoring jobs:
    On Server/Node startup, look for the following log message where 1 will vary based on configuration:
    com.sonatype.insight.brain.policy.evaluator.PolicyMonitor - insight.threads.monitor pool-size: 1

  3. Determine the time taken for Continuous Monitoring job: 
    Continuous Monitoring executes as a job by the internal quartz job scheduler, and so the thread name executing continuous monitoring will contain the word quartz. Any other thread name is not part of continuous monitoring.

    Look for the following log statements:
     

    Level Logger Message Note
    INFO com.sonatype.insight.brain.policy.evaluator.PolicyMonitor Starting policy monitoring Start of the monitoring process job for all applications and stages configured
    INFO com.sonatype.insight.brain.policy.evaluator.PolicyMonitor Policy monitoring is enabled for application '<app name>' and stage '<stage>' Start of an individual app scan
    app name - the name of an application
    stage - the policy action stage applicable
    DEBUG com.sonatype.insight.brain.policy.evaluator.PolicyMonitor Policy monitoring evaluated for application '<app name>' in <number> ms End of an individual app scan

    app name- the name of an application

    number - number of milliseconds to complete this one evaluation
    INFO com.sonatype.insight.brain.policy.evaluator.PolicyMonitor Finished policy monitoring applications in <number> ms End of the monitoring process job for all applications and stages.

    Use this reported value to know how long your monitoring job takes to finish.

    number - number of milliseconds to complete all evaluations

Adjusting Continuous Monitoring Thread Pool

If continuous monitoring job is taking longer than 24 hours to complete all scans, you can adjust the thread pool size to allow more concurrent policy evaluations.

Determine the Optimal Thread Pool Size

The optimal number of threads in the continuous monitoring job pool can be roughly calculated by this formula:

OptimalThreadPoolSize = ceiling (TotalTimeInMs / ( 20x60x60x1000 ) )

Here is a Bash and Zsh Compatible Function implementing this formula:
 

calculate_thread_pool_size() {
  local log_line="$1"
  local buffer_hours=4
  local total_hours=24
  local allowed_hours=$((total_hours - buffer_hours))
  local allowed_time_ms=$((allowed_hours * 60 * 60 * 1000))
  local total_scan_time_ms
  local pool_size

  # Use grep to extract scan time in ms (portable across Bash and Zsh)
  total_scan_time_ms=$(echo "$log_line" | grep -oE '[0-9]+[[:space:]]*ms' | grep -oE '[0-9]+')

  if [[ -z "$total_scan_time_ms" ]]; then
    echo "Error: Could not extract scan time from log line."
    return 1
  fi

  # Perform ceiling division safely
  pool_size=$(( (total_scan_time_ms + allowed_time_ms - 1) / allowed_time_ms ))

  echo "Total scan time: ${total_scan_time_ms} ms"
  echo "Allowed scan time: ${allowed_hours} hours (${allowed_time_ms} ms)"
  echo "Optimal thread pool size: $pool_size"
  echo "java argument: -Dinsight.threads.monitor=$pool_size"
}

Explanation

TotalTimeInMs = The time it currently takes to scan with a single thread ( default) or the accumulated serial workload (i.e., not yet parallelized). See "Finished policy monitoring applications in" log statement

20 = hrs target to complete all scans within 24 hours - 4 hr buffer

60 × 60 × 1000 = 1hr in milliseconds

ceiling = function round UP to the nearest whole number

IMPORTANT

This formula assumes that Server CPU and network resources are available to handle the concurrency, and linear scalability (i.e., perfect parallelism and no contention).

Example

We examine log files and find this log line:

Finished policy monitoring applications in 437824876 ms

Use our Shell Script function:

> log="Finished policy monitoring applications in 437824876 ms"
> calculate_thread_pool_size "$log"
Total scan time: 437824876 ms
Allowed scan time: 20 hours (72000000 ms)
Optimal thread pool size: 7
java argument: -Dinsight.threads.monitor=7

Adjusting the Thread Pool Size

The thread pool adjustment can only be made using a java system property and must be specified as a java argument on each IQ Server HA Node if using HA.

  • Java System Property: insight.threads.monitor=x
  • "x" is the number of threads to participate in continuous monitoring
  • 1 is the minimum value for this property
  • 20 is the maximum value for this property
  • the property is read on node startup only and cached in memory - a log statement can help you verify if the property has been adjusted. 1 is the default pool size
    com.sonatype.insight.brain.policy.evaluator.PolicyMonitor - insight.threads.monitor pool-size: 1

 

Related articles