patch-1
@reggi/path-to-regexp
dependabot/npm_and_yarn/main/copy-to-clipboard-4.0.2
dependabot/npm_and_yarn/main/eslint-10.4.0
dependabot/npm_and_yarn/main/npmcli/eslint-config-7.0.0
dependabot/npm_and_yarn/main/proc-log-7.0.0
dependabot/npm_and_yarn/npm_and_yarn-826852524d
dependabot/npm_and_yarn/npm_and_yarn-ab9a7f4bc2
deprecate-totp-2fa
dhei/classic-tokens
gat-bypass-2fa-docs
jpg619/fix-accessibility-content-flow
jpg619/version-bump-tar-2
kartykp/gat-bypass-2fa-docs
kartykp/upgrade-path-to-regex
main
maitxn/version-bump-tar
patch-1
reggi/cache-based-on-version
reggi/dev-engines
reggi/fix-transform-prettier
reggi/overrides
update-search-sensitivity
| 1 | --- |
| 2 | title: Crawler policy |
| 3 | edit_on_github: false |
| 4 | --- |
| 5 | |
| 6 | npm's full public dataset is available via the [public registry](https://docs.npmjs.com/misc/registry). Using CouchDB replication, you can get a full copy of all metadata, and it is acceptable within our terms of use to download copies of tarballs for inspection or experimentation. |
| 7 | |
| 8 | npm's [website](https://www.npmjs.com) also has package metadata available. We allow this content to be indexed by commercial crawlers such as GoogleBot. At our discretion, we also allow experimental crawlers to access the site, as long as they keep their request velocity to 1 request per second or less. At that velocity, indexing all packages would take 3 days, so if you want a full copy of our metadata it is always going to be faster to access the data via replication, which takes only an hour or two to provide full data and will thereafter automatically stay in sync. |
| 9 | |
| 10 | If you do not wish to install CouchDB to manage replication, we provide [open source software](https://github.com/npm/concurrent-couch-follower) that makes it easy to sync to the registry's public feed. |
| 11 | |
| 12 | If you attempt to access package metadata by high-velocity crawling of the npm website, we reserve the right to rate-limit or ban your IP, user-agent or both. |