Nexus can be configured to make use of Active Directory nested groups if you use a special object ID, LDAP_MATCHING_RULE_IN_CHAIN.
To do this, you need to configure Nexus to use a static group element mapping, and add the filter ":1.2.840.1135126.96.36.1991:" to the group member attribute. See here for more information about this rule.
Note that checking "group subtree" is also necessary.
See below for an example configuration. Note that the user and group mappings shown below are examples, your Active Directory server's mappings may be different depending on how your administrator has configured it.