You'll need to provide an implementation of the org.sonatype.security.usermanagement.UserManager API (you can extend AbstractReadOnlyUserManager if you just want to lookup users and not edit/add them).
Add code to return populated User objects on request, use the same @Named value as your realm (best practice is to use the same name as your realm) - this will then get picked up along with all the other UserManager implementations which are then aggregated in Nexus.
Override getAuthenticationRealmName and return your realm id, and this is used to associate the usermanager with the realm behind the scenes.
For nexus 2.8/2.9 you need to build your plugin with nexus-plugin-bundle-maven-plugin 1.2. See the example projects for examples:
A shipped example realm implementation is the rutauth realm. You can use the various branches to check for any differences that were required as the versions of Nexus changed:
https://github.com/sonatype/nexus-oss/tree/nexus-2.7.x/plugins/security/nexus-rutauth-plugin https://github.com/sonatype/nexus-oss/tree/nexus-2.8.x/plugins/security/nexus-rutauth-plugin https://github.com/sonatype/nexus-oss/tree/nexus-2.9.x/plugins/security/nexus-rutauth-plugin