Why does npm client need access to URLs other than my private registry?

Problem

After configuring a proxy to a remote npm registry, you may discover that there are still some requests that npm issues to the public internet instead of your private Nexus server.

What causes this and how can I force npm client to communicate only with Nexus?

Cause

This is a known design issue with the npm client and package specification by the npmjs.org authors, and there are several cases where this happens.

Sometimes package publishers include dependencies which reference download URLs to places like github, instead of a dependency which can be resolved at the same source registry that is being proxied.

Another situation where external access may be required is when the package author has used npm shrinkwrap to store the resolved URL in a package npm-shrinkwrap.json file and then published this package to the remote registry. When the npm client encounters the resolved field, it tries to contact that URL to download packages, instead of the configured registry in your .npmrc file. If one chooses to bypass the shrinkwrap file, then they might also lose the specific dependency versions that the package depends on.

Another case where external access is attempted by the npm client, is specific to post install scripts executed by certain npm packages themselves, like PhantomJS and node-saas

Packages like the phantomjs npm are distinct from the actual framework downloadable from phantomjs.org. After it is downloaded through Nexus by the npm command line tool, a post install script is automatically run that attempts to download the complete framework from https://bitbucket.org/ariya/phantomjs/downloads . Your npm client host may be behind a firewall blocking access to this remote site.

These problems are created by the authors of npm shrinkwrap and npm client, or well meaning package authors. Sonatype has filed and commented on several issues with the official npm maintainers about these problems, yet the issues remain despite the support of many community members.

https://github.com/npm/npm/issues/3581#issuecomment-58516251

https://github.com/npm/npm/issues/6444

https://github.com/npm/npm/issues/6445

In summary, the design decisions of the npm repository format authors complicate the adoption of private registries like Nexus. We believe that a repository client should have full control of where packages come from, and that a repository format should instead focus on providing a unique identifier to a component that is source URL independent. 

Workaround

Users of npm client are invited to comment on existing issues, or file new issues with npm, if you have an opinion about the design decisions leading to the mentioned problems.

The publishers of the npmjs blog have posted an article on dealing with problematic dependencies. Please refer to that article for their latest guidance on working around these common problems.

The following advice has been reported to work for some:

In the case where the package includes a npm-shrinkwrap.json file, you can one time ignore this file by using --no-shrinkwrap argument with your npm install command. Keep in mind this will also ignore explicit dependency versions specified in this file.

The workaround for permanently bypassing embedded package.json URLs ends up being a variation of:

  1. Download the original package and extract it.
  2. Edit the package.json file manually or use a tool like npm shonkwrap to remove the shrinkwrap resolved URLs in the package npm-shrinkwrap.json file
  3. Republish the edited package to a private hosted npm repository in Nexus.
  4. Possibly publish additional dependency packages to your private hosted registry.

In the PhantomJS package case, one can either download the binary and manually install it on the PATH or you can override the location where the installer gets it's binary to an internal server URL you host. For the second option, you could even create a hosted site repository in Nexus and put the downloads there.

For node-saas post install script, you may be able to override binary download location that is used.

Related Issues

Why does Nexus download npm packages from sites other than the configured remote URL?

Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk