There is a 4Gb limit imposed by the Java Service Wrapper watchdog process used by Nexus.
If you need more memory, comment out the
wrapper.java.maxmemory line in
<nexus_root>/bin/jsw/conf/wrapper.conf and add a line to the wrapper.java.additional arguments. Example:
Make sure the sequence number (shown here as "4") is consecutive with the other additional argument lines.
Note: It is very unlikely that you actually need more than 4Gb of heap space.