Visit my.sonatype.com for documentation on Nexus Repository version 2.
There is a 4 gb limit imposed by the Java Service Wrapper watchdog process used by Nexus Repository 2.
If you need more memory, comment out the wrapper.java.maxmemory line in <nexus_root>/bin/jsw/conf/wrapper.conf and add a line to the wrapper.java.additional arguments. Example:
wrapper.java.additional.4=-Xmx5120m
Make sure the sequence number (shown here as "4") is consecutive with the other additional argument lines.
Note: It is very unlikely that you actually need more than 4 gb of heap space.