Add info to native packages docs about mirroring our repos. (#16069)
* Add info to native packages docs about mirroring our repos. * Add note about disabling robots.txt handling in mirroring tools.
Austin S. Hemmelgarn committed
Oct 18, 2023 at 08:26 UTC
a155cca000b0dd1ceeeb8f99705f059c37970ba6
1 file changed
+44
-3
packaging/installer/methods/packages.md
+44
-3
@@ -23,10 +23,10 @@ and fail if it cannot do so.
23
24
25
> ### Note
26
->
26
+>
27
> In July 2022, we switched hosting of our native packages from Package Cloud to self-hosted repositories.
28
-> We still maintain the Package cloud repositories, but they are not guaranteed to work and may be removed
29
-> without prior warning.
28
+> We still maintain the Package cloud repositories, but they are not guaranteed to work and may be removed
29
+> without prior warning.
30
>
31
> When selecting a repository configuration package, note that the version 2 packages provide configuration for
32
> our self-hosted repositories, and then version 1 packages provide configuration for Package Cloud.
@@ -99,3 +99,44 @@ If you are explicitly configuring a system to use our repositories, the recommen
99
appropriate repository configuration package from https://repo.netdata.cloud/repos/repoconfig and install it
100
directly on the target system using the system package manager. This will ensure any packages needed to use the
101
repository are also installed, and will help enable a seamless transition if we ever need to change our infrastructure.
102
+
103
+## Local mirrors of the official Netdata repositories
104
+
105
+Local mirrors of our official repositories can be created in one of two ways:
106
+
107
+1. Using the standard tooling for mirroring the type of repository you want a local mirror of, such as Aptly for
108
+ APT repositories, or reposync for RPM repositories. For this approach, please consult the documentation for
109
+ the specific tool you are using for info on how to mirror the repositories.
110
+2. Using a regular website mirroring tool, such as GNU wget’s `--mirror` option. For this approach, simply point
111
+ your mirroring tool at `https://repo.netdata.cloud/repos/`, and everything should just work.
112
+
113
+We do not provide official support for mirroring our repositories,
114
+but we do have some tips for anyone looking to do so:
115
+
116
+- Our `robots.txt` file explicitly disallows indexing, so if you’re using a regular website mirroring tool,
117
+ you wil need to tell it to ignore `robots.txt` (for example, if using GNU wget, add `-e robots=off` to the
118
+ options you pass) to ensure that it actually retrieves everything.
119
+- Excluding special cases of caching proxies (such as apt-cacher-ng), our repository configuration packages _DO NOT_
120
+ work with custom local mirrors. Thus, you will need to manually configure your systems to use your local mirror.
121
+- Packages are published as they are built, with 64-bit x86 packages being built first, followed by 32-bit x86,
122
+ and then non-x86 packages in alphabetical order of the CPU architecture. Because of the number of different
123
+ packages being built, this means that packages for a given nightly build or stable release are typically published
124
+ over the course of a few hours, usually starting about 15-20 minutes after the build or release is started.
125
+- Repository metadata is updated every hour on the hour, and the process may take anywhere from a few seconds to
126
+ more than 20 minutes. Because of this, it makes little sense to sync your mirror more frequently than once an hour,
127
+ and it’s generally preferred to start syncing at least 30 minutes into the hour.
128
+- A full mirror of all of our repositories currently requires up to 100 GB of storage space, though the exact
129
+ amount of space needed fluctuates over time. Because of this, users seeking to mirror our repositories are
130
+ encouraged to mirror only those repositories they actually need instead of mirroring everything.
131
+- If syncing daily (or less frequently), some time between 05:00 and 08:00 UTC each day is usually the saftest
132
+ time to do so, as publishing nightly packages will almost always be done by this point, and publishing of stable
133
+ releases typically happens after that time window.
134
+- If you intend to use our existing GPG signatures on the repository metadata and packages, you probably also want
135
+ a local copy of our public GPG key, which can be fetched from `https://repo.netdata.cloud/netdatabot.gpg.key`.
136
+
137
+## Public mirrors of the official Netdata repositories
138
+
139
+There are no official public mirrors of our repositories.
140
+
141
+If you wish to provide a public mirror of our official repositories, you are free to do so, but we kindly ask that
142
+you make it clear to your users that your mirror is not an official mirror of our repositories.