@cryptotaxi247 / infra / commits / beeae7d0

treefmt: add deno for markdown, json and yaml

Jörg Thalheim committed Dec 1, 2024 at 20:35 UTC beeae7d0233d4aaaa77ed378b884fdb3e46d60f3

30 files changed +931 -775

.github/ISSUE_TEMPLATE/feature_request.md

+1 -2

@@ -4,12 +4,11 @@ about: Suggest an improvement for this project

4 title: ''

5 labels: enhancement

6 assignees: ''

7 -

7 ---

9 **Is your feature request related to a problem? Please describe.**

12 -<!--

11 +<!--

12 A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

13 -->

.github/ISSUE_TEMPLATE/service_disruption.md

-2

@@ -4,7 +4,6 @@ about: Use this to report service instabilities

4 title: '<service-name>: '

5 labels: bug

6 assignees: ''

7 -

7 ---

9 **Affected service**

@@ -18,4 +17,3 @@ assignees: ''

17 **System information**

19

21 -

.github/workflows/ci.yml

+12 -12

@@ -16,10 +16,10 @@ jobs:

16 nix-build:

17 runs-on: ubuntu-latest

18 steps:

19 - - uses: actions/checkout@v4

20 - - uses: cachix/install-nix-action@v30

21 - - uses: DeterminateSystems/magic-nix-cache-action@v8

22 - - run: nix run github:Mic92/nix-fast-build -- --no-nom

19 + - uses: actions/checkout@v4

20 + - uses: cachix/install-nix-action@v30

21 + - uses: DeterminateSystems/magic-nix-cache-action@v8

22 + - run: nix run github:Mic92/nix-fast-build -- --no-nom

23 # all builds combined consume too much disk space... we should soon switch to something else

24 nixos:

25 runs-on: ubuntu-latest

@@ -30,10 +30,10 @@ jobs:

30 #machine: [caliban, umbriel]

31 machine: [caliban]

32 steps:

33 - - uses: actions/checkout@v4

34 - - uses: cachix/install-nix-action@v30

35 - - uses: DeterminateSystems/magic-nix-cache-action@v8

36 - - run: nix build '.#nixosConfigurations."${{ matrix.machine }}.nixos.org".config.system.build.toplevel'

33 + - uses: actions/checkout@v4

34 + - uses: cachix/install-nix-action@v30

35 + - uses: DeterminateSystems/magic-nix-cache-action@v8

36 + - run: nix build '.#nixosConfigurations."${{ matrix.machine }}.nixos.org".config.system.build.toplevel'

37 nix-darwin:

38 runs-on: macos-latest

39 strategy:

@@ -41,7 +41,7 @@ jobs:

41 # Doesn't seem that x86_64 is still in use?

42 machine: [arm64]

43 steps:

44 - - uses: actions/checkout@v4

45 - - uses: cachix/install-nix-action@v30

46 - - uses: DeterminateSystems/magic-nix-cache-action@v8

47 - - run: nix build '.#darwinConfigurations."${{ matrix.machine }}".config.system.build.toplevel'

44 + - uses: actions/checkout@v4

45 + - uses: cachix/install-nix-action@v30

46 + - uses: DeterminateSystems/magic-nix-cache-action@v8

47 + - run: nix build '.#darwinConfigurations."${{ matrix.machine }}".config.system.build.toplevel'

README.md

+9 -8

@@ -3,13 +3,12 @@

3 This repository contains all the hardware configuration for the nixos project

4 infrastructure.

6 -All the hosts are currently managed using NixOps. Some of the infrastructure

7 -is managed using Terraform. There are still a lot of things configured

8 -manually.

6 +All the hosts are currently managed using NixOps. Some of the infrastructure is

7 +managed using Terraform. There are still a lot of things configured manually.

9 ## Docs

12 -* [Resources inventory](docs/inventory.md)

11 +- [Resources inventory](docs/inventory.md)

13 ## Team

@@ -29,9 +28,11 @@ All the members should be watching this repository for changes.

29 ## Regular catch up

32 -We meet regularly over Jitsi to hash some issues out. Sometimes it helps to have dedicated focus and higher communication bandwidth.

31 +We meet regularly over Jitsi to hash some issues out. Sometimes it helps to have

32 +dedicated focus and higher communication bandwidth.

34 -It started Thursday, January 11, 2024, at 6 pm CET (UTC+1), and then repeats every two weeks, on Thursdays at 6 pm CET.

34 +It started Thursday, January 11, 2024, at 6 pm CET (UTC+1), and then repeats

35 +every two weeks, on Thursdays at 6 pm CET.

37 <a target="_blank" href="https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MDVjdjNpOG5qazhscjlna3Mxcmw0aHVzODIgam9uYXNAbnVtdGlkZS5jb20&tmsrc=jonas%40numtide.com"><img border="0" src="https://www.google.com/calendar/images/ext/gc_button1_en.gif"></a>

@@ -39,7 +40,7 @@ Location: <https://jitsi.lassul.us/nixos-infra>

41 ## Reporting issues

42 -If you experience any issues with the infrastructure, please [post a new issue

43 -to this repository][1].

43 +If you experience any issues with the infrastructure, please

44 +[post a new issue to this repository][1].

46 [1]: https://github.com/NixOS/nixos-org-configurations/issues/new

build/haumea/zrepl.yml

+3 -3

@@ -5,10 +5,10 @@

5 global:

6 logging:

7 - type: "stdout"

8 - level: "error"

8 + level: "error"

9 format: "human"

10 - type: "syslog"

11 - level: "info"

11 + level: "info"

12 format: "logfmt"

14 # mostly from https://blog.lenny.ninja/zrepl-on-rsync-net.html

@@ -17,7 +17,7 @@ jobs:

17 type: sink

18 serve:

19 type: stdinserver

20 - client_identities: [ haumea ]

20 + client_identities: [haumea]

21 recv:

22 placeholder:

23 encryption: off

build/nginx-error-pages/503.html

+24 -24

@@ -1,29 +1,29 @@

1 <!DOCTYPE html>

3 <html>

4 + <head>

5 + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

6 + <title>Hydra is down</title>

7 + <style type="text/css" media="screen">

8 + body {

9 + font-family: Helvetica, Arial, sans-serif;

10 + color: rgba(0, 0, 0, 0.7);

11 + }

12 + </style>

13 + </head>

5 -<head>

6 - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

7 - <title>Hydra is down</title>

8 - <style type="text/css" media="screen">

9 - body {

10 - font-family: Helvetica, Arial, sans-serif;

11 - color: rgba(0, 0, 0, 0.7);

12 - }

13 - </style>

14 -</head>

15 -

16 -<body>

17 - <center>

18 - <img src="/apache-errors/warning.png" alt="Warning" />

19 - <p>Looks like Hydra is having some problems. Sorry about that!</p>

20 - <p style="font-size: 90%;">

21 - <a href="https://nixos.org/">NixOS Homepage</a> |

22 - <a href="https://monitoring.nixos.org/prometheus/alerts">System Alerts</a> |

23 - <a href="https://monitoring.nixos.org/grafana/">Dashboards</a> |

24 - <a href="https://github.com/NixOS/nixpkgs/labels/infrastructure">Related Issues</a>

25 - </p>

26 - </center>

27 -</body>

28 -

15 + <body>

16 + <center>

17 + <img src="/apache-errors/warning.png" alt="Warning" />

18 + <p>Looks like Hydra is having some problems. Sorry about that!</p>

19 + <p style="font-size: 90%">

20 + <a href="https://nixos.org/">NixOS Homepage</a> |

21 + <a href="https://monitoring.nixos.org/prometheus/alerts"

22 + >System Alerts</a> |

23 + <a href="https://monitoring.nixos.org/grafana/">Dashboards</a> |

24 + <a href="https://github.com/NixOS/nixpkgs/labels/infrastructure"

25 + >Related Issues</a>

26 + </p>

27 + </center>

28 + </body>

29 </html>

build/rhea/install.md

+22 -16

@@ -1,13 +1,14 @@

1 # Setup

3 ## Switch to UEFI

4 -First submit a support ticket asking them to enable UEFI.

5 -See: https://docs.hetzner.com/robot/dedicated-server/operating-systems/uefi/

4 +

5 +First submit a support ticket asking them to enable UEFI. See:

6 +https://docs.hetzner.com/robot/dedicated-server/operating-systems/uefi/

8 # Correct the NVMe namespace's block size

9 -Verify the NVMe disks are formatted at the namespace level with 4096 blocks.

10 -See https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Hardware.html#nvme-low-level-formatting

10 +Verify the NVMe disks are formatted at the namespace level with 4096 blocks. See

11 +https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Hardware.html#nvme-low-level-formatting

13 This disk's LBA is 512:

@@ -236,7 +237,8 @@ nixos-generate-config --root /mnt

237 In the `configuration.nix`:

238

239 1. Add `hetzner.nix` to the list of `imports` at the top.

239 -2. Add an authorized key and enable SSH. This will be removed later when it is imported into NixOps, so it is just for bootstrapping:

240 +2. Add an authorized key and enable SSH. This will be removed later when it is

241 + imported into NixOps, so it is just for bootstrapping:

242

243 ```

244 services.openssh.enable = true;

@@ -245,24 +247,28 @@ users.users.root.openssh.authorizedKeys.keys = [ "ssh-..." ];

247

248 ### Hardware Configuration Changes

249

248 -Edit `hardware-configuration.nix` and change the fileSystems value for `/nix/var/nix` to make it required for boot:

250 +Edit `hardware-configuration.nix` and change the fileSystems value for

251 +`/nix/var/nix` to make it required for boot:

252

253 ```nix

251 - fileSystems."/nix/var/nix/db" =

252 - { device = "rpool/local/nix/db";

253 - fsType = "zfs";

254 - neededForBoot = true;

255 - };

254 +fileSystems."/nix/var/nix/db" =

255 + { device = "rpool/local/nix/db";

256 + fsType = "zfs";

257 + neededForBoot = true;

258 + };

259 ```

260

261 ### Hetzner.nix

262

263 Then create a file, `hetzner.nix`.

264

262 -* The all-zeros hostId is fine, though I generated one with `head -c4 /dev/urandom | od -A none -t x4`

263 -* The `enp7s0` and `MACAddress` value I got from `ip addr`

264 -* The IP addresses and gateways I got from the Robot webpage under the IPs tab, hovering over the IPv4 and IPv6 addresses.

265 -* Thee DNS resolvers I got from https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/

265 +- The all-zeros hostId is fine, though I generated one with

266 + `head -c4 /dev/urandom | od -A none -t x4`

267 +- The `enp7s0` and `MACAddress` value I got from `ip addr`

268 +- The IP addresses and gateways I got from the Robot webpage under the IPs tab,

269 + hovering over the IPv4 and IPv6 addresses.

270 +- Thee DNS resolvers I got from

271 + https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/

272

273 ```nix

274 {

@@ -302,4 +308,4 @@ Then run:

308

309 ```

310 nixos-install -I nixpkgs=channel:nixos-21.11

305 -```

\ No newline at end of file

311 +```

docs/inventory.md

+49 -60

@@ -3,14 +3,15 @@

3 This is the current list of hardware and services that everyone has access to.

5 # Accounts

6 +

7 ## GitHub

9 owner: @edolstra @domenkozar @garbas @grahamc @rbvermaa

11 ## Domains

12 -owner: @edolstra

13 -* nixos.org - https://www.uniteddomains.com/

13 +- owner: @edolstra

14 +- nixos.org - https://www.uniteddomains.com/

16 ## DNS

@@ -20,25 +21,25 @@ Managed by Netlify.

22 ## AWS account

23 -owner: Infor

24 -alias: lb-nixos

25 -access: @rbvermaa and @edolstra

24 +- owner: Infor

25 +- alias: lb-nixos

26 +- access: @rbvermaa and @edolstra

28 ## Packet.net

29 -owner: @grahamc

30 +- owner: @grahamc

32 ## Hetzner Cloud

33 -owner: Graham

34 -(for ofborg)

34 +- owner: Graham

35 +- (for ofborg)

37 ## IRC logging bot

38 -owner: @samueldr

39 -url: https://logs.nix.samueldr.com/nixos/

40 -nick: <code>{\`-\`}</code>

41 -config: https://gitlab.com/samueldr.nix/overlays/irclogger

39 +- owner: @samueldr

40 +- url: https://logs.nix.samueldr.com/nixos/

41 +- nick: <code>{\`-\`}</code>

42 +- config: https://gitlab.com/samueldr.nix/overlays/irclogger

44 ## nix.ci

@@ -50,9 +51,9 @@ hosted on Packet.

52 ## arch64 community builder

53 -owner: @grahamc

54 -access: community members that have asked access to it

55 -host: Packet

54 +- owner: @grahamc

55 +- access: community members that have asked access to it

56 +- host: Packet

58 lots of cores to build for the aarch64 platform

@@ -64,7 +65,6 @@ owner: @davidak

66 owner: Christine?

67 -

68 ## nixcon2018.org

70 owner: @zimbatm

@@ -75,22 +75,18 @@ access: see https://wiki.nixos.org/wiki/NixOS_Wiki:About

76 ## Twitter accounts

78 -**nixpkg**

79 -owner: Graham

78 +**nixpkg** owner: Graham

81 -**nixos_org**

82 -owner: Rob Vermaas

80 +**nixos_org** owner: Rob Vermaas

84 -**nixcon2017**

85 -owner: Christine?

86 -

87 -**nixcon2018**

88 -owner: zimbatm

82 +**nixcon2017** owner: Christine?

84 +**nixcon2018** owner: zimbatm

86 ## IRC

93 -Group registration on FreeNode. Eelco and Graham can get OP on all channels about NixOS.

88 +Group registration on FreeNode. Eelco and Graham can get OP on all channels

89 +about NixOS.

91 The group owns:

@@ -104,55 +100,46 @@ The group owns:

100

101 `**#nixos-dev**`

102

107 -`#``**nixos**`

108 -

109 -1 niksnut +AFRefiorstv [modified ? ago]

110 -17:30 2 goodwill +o [modified 3y 36w 6d ago] -

111 -17:30 3 kmicu +o [modified 2y 32w 5d ago] long time member - left 4 months ago

112 -17:30 4 gchristensen +o [modified 1y 37w 1d ago]

113 -

114 -`**#nixos-borg**`

115 -`**#nixos-aarch64**`

116 -`**#nix-darwin**`

117 -`#nixos-chat`

118 -`**#nix-core**`

119 -`**#nixos-security**`

120 -`**#nixos-bots**`

121 -`**#nixos-docs**`

122 -`**#nixos-wiki**`

123 -`**#nixos-on-your-router**`

124 -

103 +`#` **nixos**``

104

105 +- 1 niksnut +AFRefiorstv [modified ? ago]

106 +- 17:30 2 goodwill +o [modified 3y 36w 6d ago] -

107 +- 17:30 3 kmicu +o [modified 2y 32w 5d ago] long time member - left 4 months ago

108 +- 17:30 4 gchristensen +o [modified 1y 37w 1d ago]

109

110 +`**#nixos-borg**` `**#nixos-aarch64**` `**#nix-darwin**` `#nixos-chat`

111 +`**#nix-core**` `**#nixos-security**` `**#nixos-bots**` `**#nixos-docs**`

112 +`**#nixos-wiki**` `**#nixos-on-your-router**`

113

114 ## cachix.org

115

116 owner: Domen

117

118 # Hardware

119 +

120 ## On Packet.net

121

122 owner: Graham

123

137 -

124 2 builders: aarch64 packet type 2 : for hydra

125

140 -1 aarch64 for ofborg *and* community use

126 +1 aarch64 for ofborg _and_ community use

127

128 ## Hetzner:

129

130 owner: Eelco and Rob, owned by the NixOS Foundation

131

146 -“chef”: runs hydra.nixos.org, postgresql database, queue runner, hydra provisioner. binary cache signing keys.

132 +“chef”: runs hydra.nixos.org, postgresql database, queue runner, hydra

133 +provisioner. binary cache signing keys.

134

148 -monitoring:

149 -**DataDog, accessible by Eelco (and Rob?) (Amine?) on the Infor account**

135 +monitoring: **DataDog, accessible by Eelco (and Rob?) (Amine?) on the Infor

136 +account**

137

138 ## Mac Minis at Hetzner Cloud

139

153 -owner: the NixOS Foundation

154 -access: Cole-h & Hexa

155 -role: build machines

140 +- owner: the NixOS Foundation

141 +- access: Cole-h & Hexa

142 +- role: build machines

143

144 Current machine names:

145

@@ -164,9 +151,9 @@ Current machine names:

151

152 ## Mac Minis at Graham's house

153

167 -owner: the NixOS Foundation

168 -access: Cole-h

169 -role: build machines

154 +- owner: the NixOS Foundation

155 +- access: Cole-h

156 +- role: build machines

157

158 - arm64:

159 - cosmic-stud

@@ -174,12 +161,13 @@ role: build machines

161 - quality-ram

162 - becoming-hyena

163

177 -There are also x86_64 mac minis, but they are offline because they produce too much heat.

164 +There are also x86_64 mac minis, but they are offline because they produce too

165 +much heat.

166

167 ## Mac Stadium

168

181 -owner: MacStadium and rented to daniel peebles or the foundation?

182 -role: build machines

169 +- owner: MacStadium and rented to daniel peebles or the foundation?

170 +- role: build machines

171

172 Eelco had a root password

173

@@ -193,7 +181,8 @@ owner: LogicBlox EC2 instance

181

182 deployed from Eelco’s laptop

183

196 -runs the website

197 -runs the channel mirror script, systemd services with timers, updates /releases buckets and the nixpkgs-channels repository (repo: nixos-channel-scripts)

184 +runs the website runs the channel mirror script, systemd services with timers,

185 +updates /releases buckets and the nixpkgs-channels repository (repo:

186 +nixos-channel-scripts)

187

188 The tarball mirror script is running from that machine.

docs/meeting-notes/2024-01-11.md

+17 -14

@@ -6,13 +6,13 @@ Participants: delroth, hexa, raitobezarius, vcunat, zimbatm

7 ## [zimbatm] Presentation

9 -- At NixCon, we added new people to the team, but we were not able to give space to those new

10 - people, with this in mind, I would like to dedicate one hour per week or two weeks where I can

11 - unblock the infrastructure matters.

12 -- I don’t know what people are interested in, I believe this is a volunteer ecosystem and you should

13 - work on what you would like to work on.

14 -- We have big challenges in front of us, e.g. the cache situation, with a new team, maybe we can

15 - tackle those bigger challenges.

9 +- At NixCon, we added new people to the team, but we were not able to give space

10 + to those new people, with this in mind, I would like to dedicate one hour per

11 + week or two weeks where I can unblock the infrastructure matters.

12 +- I don’t know what people are interested in, I believe this is a volunteer

13 + ecosystem and you should work on what you would like to work on.

14 +- We have big challenges in front of us, e.g. the cache situation, with a new

15 + team, maybe we can tackle those bigger challenges.

17 ## Round of intros

@@ -34,7 +34,8 @@ Skipped in these edited notes.

34 - delroth/hexa are in favor of self-hosting.

35 - But we need the database dump from EMS.

36 - hexa to prepare the config for this, delroth can act as backup/fallback.

37 -- Fallback: we can always pay the $1200 (excl. VAT) for renewing the 1 year plan.

37 +- Fallback: we can always pay the $1200 (excl. VAT) for renewing the 1 year

38 + plan.

40 ## [hexa] Moving NGI out of nixos-org-configurations

@@ -46,16 +47,18 @@ Skipped in these edited notes.

48 ## Builders

49 -- Context: various cost reduction efforts need to happen on the Hydra/ofborg builders infra.

50 +- Context: various cost reduction efforts need to happen on the Hydra/ofborg

51 + builders infra.

52 - There might be the possibility to get Hetzner to sponsor one more machine.

51 -- [delroth] Pretty sure we are not using our build resources efficiently as it is (queue-runner

52 - bottleneck)

53 +- [delroth] Pretty sure we are not using our build resources efficiently as it

54 + is (queue-runner bottleneck)

55 - [vcunat] xz compression is the main problem

56 - [zimbatm] We should properly analyze where the bottlenecks are.

58 ## Backups

60 - We are not doing proper backups of the NixOS infra.

59 -- There is an rsync.net account where the Hydra database gets backed up to, at least.

60 -- Julien's vaultwarden PR is currently blocked by this, we're getting backup storage space from

61 - Hetzner (storage boxes).

61 +- There is an rsync.net account where the Hydra database gets backed up to, at

62 + least.

63 +- Julien's vaultwarden PR is currently blocked by this, we're getting backup

64 + storage space from Hetzner (storage boxes).

docs/meeting-notes/2024-01-25.md

+8 -5

@@ -4,8 +4,8 @@

5 - Configuration hasn’t been written yet, hexa might get it done this week.

6 - When will we get the data?

7 - - Graham still holding it until it can get cleaned up (removing private user data). Board set a

8 - deadline during the last meeting.

7 + - Graham still holding it until it can get cleaned up (removing private user

8 + data). Board set a deadline during the last meeting.

9 - We could talk to EMS directly, to get the account handed over

10 - We want ~10 days to do the migration (so: we want the data before Feb 7th)

@@ -33,9 +33,12 @@

33 - Advanced communication will be sent out

34 - Build list of store paths we want to keep and configure gc root for them

35 - Plan is to keep all FODs

36 - - Make store paths that are about to get deleted unavailable prior to deletion

36 + - Make store paths that are about to get deleted unavailable prior to

37 + deletion

38 - Potentially move parts of the cache to Hetzner

39 - delroth has capacity to look into this in 2024/02

39 - - Needs a service to decide, where (S3 or Hetzner) the request would need to go

40 + - Needs a service to decide, where (S3 or Hetzner) the request would need to

41 + go

42 - Logic could be installed at fastly, to try hetzner first, fallback to s3

41 - - Service is in the critical path, currently fastly/s3 solve availability for us

43 + - Service is in the critical path, currently fastly/s3 solve availability for

44 + us

docs/meeting-notes/2024-02-08.md

+59 -44

@@ -1,6 +1,7 @@

1 # 2024-02-08

3 -Attendees: delroth, hexa, JulienMalka, lheckemann, raitobezarius, vcunat, zimbatm

3 +Attendees: delroth, hexa, JulienMalka, lheckemann, raitobezarius, vcunat,

4 +zimbatm

6 ## [hexa, delroth] EMS Migration

@@ -9,34 +10,37 @@ Context: https://github.com/NixOS/infra/issues/325

10 - PR for Synapse and its dependencies is up.

11 - https://github.com/NixOS/infra/pull/336

12 - [Julien] What's the status of the backup module?

12 - - Split off into its own PR and merged already: https://github.com/NixOS/infra/pull/345

13 + - Split off into its own PR and merged already:

14 + https://github.com/NixOS/infra/pull/345

15 - raito and Ron met with Matrix / EMS folks at FOSDEM 2024

14 - - They have scripts for GDPR compliance (user data purge), but we need to ask them by email.

16 + - They have scripts for GDPR compliance (user data purge), but we need to ask

17 + them by email.

18 - Then we can get a clean DB dump, presumably without user data.

16 - - Not sure whether we sent an email or not. But Graham might be in contact directly, and EMS folks

17 - made him an offer to do the data deletion.

19 + - Not sure whether we sent an email or not. But Graham might be in contact

20 + directly, and EMS folks made him an offer to do the data deletion.

21 - Worst case Graham/DetSys will pay for the extension of the EMS plan.

19 - - Probably no hurry anymore from the infra side. Foundation board is monitoring this to make sure

20 - we have a solution at some point.

22 + - Probably no hurry anymore from the infra side. Foundation board is

23 + monitoring this to make sure we have a solution at some point.

25 ## [delroth] Should we publish these notes more widely?

27 - There is a trend towards publishing notes on Discourse, etc. for visibility.

25 -- [delroth] My thoughts: we should archive (edited) notes in Git somewhere in our docs/ folder,

26 - update a Discourse thread every 2 weeks.

28 +- [delroth] My thoughts: we should archive (edited) notes in Git somewhere in

29 + our docs/ folder, update a Discourse thread every 2 weeks.

30 - I of course volunteer to take care of this :)

31 - Consensus: let’s do it.

33 ## [delroth] Packet/EQM access to infra-core

32 -- Our builders are very, very outdated. But risky to try and update stuff with 0 debugging

33 - capabilities.

34 -- Any reason why infra-core shouldn’t have full Packet/EQM access like we have Hetzner access?

35 +- Our builders are very, very outdated. But risky to try and update stuff with 0

36 + debugging capabilities.

37 +- Any reason why infra-core shouldn’t have full Packet/EQM access like we have

38 + Hetzner access?

39 - Not entirely clear who currently has access?

40 - [zimbatm] Got access from eelco last weekend, will delegate.

41 - [raito] Does nix-netboot-serve run on our infra?

38 - - [hexa] Yes, on eris. The images are also built from our infra, it’s a Hydra jobset. But the

39 - jobset has not successfully completed for a year.

42 + - [hexa] Yes, on eris. The images are also built from our infra, it’s a Hydra

43 + jobset. But the jobset has not successfully completed for a year.

44 - [hexa] We can update stuff, but we have no way to debug issues if we do so.

45 - zimbatm took care of it live, woo!

@@ -47,43 +51,53 @@ Context: https://github.com/NixOS/infra/issues/325

51 - In general: who should have ownership to accounts?

52 - A bunch of GH org owners for example are inactive.

53 - Not really aligned with any subgroup e.g. foundation board.

50 - - [zimbatm] I think the foundation should have access, but unfortunately the foundation also

51 - doesn’t have the best personal security to hold those credentials.

52 - - [zimbatm] Maybe it should be the infra team instead? i.e. delroth/hexa/vcunat/…

53 - - [raito] That would work too, as long as it’s active folks who can take care of day to day stuff.

54 - I don’t care that it’s specifically me, just that we don’t get blocked due to not finding an

55 - owner.

56 - - [zimbatm] I don’t feel like I can make that decision alone right now. Let’s find some kind of

57 - organization which makes sense.

54 + - [zimbatm] I think the foundation should have access, but unfortunately the

55 + foundation also doesn’t have the best personal security to hold those

56 + credentials.

57 + - [zimbatm] Maybe it should be the infra team instead? i.e.

58 + delroth/hexa/vcunat/…

59 + - [raito] That would work too, as long as it’s active folks who can take care

60 + of day to day stuff. I don’t care that it’s specifically me, just that we

61 + don’t get blocked due to not finding an owner.

62 + - [zimbatm] I don’t feel like I can make that decision alone right now. Let’s

63 + find some kind of organization which makes sense.

64 - Raito got invited into the private infra matrix channel (at least, for now)

66 ## [Julien] NixOS wiki collaboration w/ infra team

62 -- We have a bunch of candidate sysadmins in mind. Do we want to merge this into non-critical-infra?

63 -- [Julien] I’m a bit biased since I’m sitting on both sides of this discussion, but I think this

64 - would be a good onramp to bring more people into non-critical-infra.

65 -- [zimbatm] We can subdivide permissions on the Hetzner Cloud side of things, but I’m not sure

66 - whether we should share stuff further.

67 -- [hexa] They have their setup mostly figured out already, including backups. We can let them run

68 - with it for now, and we can always pick it up later.

69 -- [linus] What about inviting them to non-critical-infra and just giving them access to all the

70 - non-critical-infra? Even if they just want to maintain the wiki.

71 - - [hexa] It’s about responsible for all of it. I don’t think we should grant unneeded access.

68 +- We have a bunch of candidate sysadmins in mind. Do we want to merge this into

69 + non-critical-infra?

70 +- [Julien] I’m a bit biased since I’m sitting on both sides of this discussion,

71 + but I think this would be a good onramp to bring more people into

72 + non-critical-infra.

73 +- [zimbatm] We can subdivide permissions on the Hetzner Cloud side of things,

74 + but I’m not sure whether we should share stuff further.

75 +- [hexa] They have their setup mostly figured out already, including backups. We

76 + can let them run with it for now, and we can always pick it up later.

77 +- [linus] What about inviting them to non-critical-infra and just giving them

78 + access to all the non-critical-infra? Even if they just want to maintain the

79 + wiki.

80 + - [hexa] It’s about responsible for all of it. I don’t think we should grant

81 + unneeded access.

82 - [Julien] +1.

73 - - [delroth] I feel like if it’s official, we should treat it as such and onboard it as part of

74 - non-critical infra. Doesn’t require giving them access to everything.

75 - - [linus] If it is official, then it should be maintained by the official infra team

83 + - [delroth] I feel like if it’s official, we should treat it as such and

84 + onboard it as part of non-critical infra. Doesn’t require giving them access

85 + to everything.

86 + - [linus] If it is official, then it should be maintained by the official

87 + infra team

88 - [hexa] I think we’re mostly in agreement then.

77 -- [delroth] non-critical-infra should be restricted to the relevant directories and go through PRs

78 - for touching other stuff

89 +- [delroth] non-critical-infra should be restricted to the relevant directories

90 + and go through PRs for touching other stuff

91 - [Julien] They probably want to iterate fast in the beginning

80 - - [delroth] They should get a dedicated machine on Hetzner Cloud, that they can play with

92 + - [delroth] They should get a dedicated machine on Hetzner Cloud, that they

93 + can play with

94 - [Julien] Too much shared code will increase reliance on core infra members.

95 - [delroth] Action items

96 - Let’s give them SSH access to a Hetzner Cloud VM

84 - - Or a separate project so they get direct access to machines. Might already be done.

85 - - Let’s make sure we agree on the idea of moving this to non-critical-infra in the short/mid-term

86 - future

97 + - Or a separate project so they get direct access to machines. Might already

98 + be done.

99 + - Let’s make sure we agree on the idea of moving this to non-critical-infra in

100 + the short/mid-term future

101 - Provision DNS etc.

102

103 ## External requests

@@ -91,11 +105,12 @@ Context: https://github.com/NixOS/infra/issues/325

105 - Hydra DB access (raitobezarius)

106 - Hashing out details in https://github.com/NixOS/infra/issues/348

107 - CA derivations for Hydra (Ericson2314)

94 - - Nix 2.20 broke interop with the old Nix 2.13 we run on builders. Rolled back to 2.19.

108 + - Nix 2.20 broke interop with the old Nix 2.13 we run on builders. Rolled back

109 + to 2.19.

110 - https://github.com/NixOS/nix/issues/9961

111 - DB schema change applied.

112

113 ## Ongoing projects

114

100 -- [delroth] Hoping to complete the nixops deprecation this week. Then: core/non-critical-infra

101 - alignment.

115 +- [delroth] Hoping to complete the nixops deprecation this week. Then:

116 + core/non-critical-infra alignment.

docs/meeting-notes/2024-02-22.md

+106 -88

@@ -11,109 +11,122 @@ Attendees: delroth, edolstra, hexa, JulienMalka, raitobezarius, vcunat, zimbatm

12 - How do we backup haumea, long term?

13 - borgbackup isn't really a good fit for a 500GB Postgres DB.

14 - - Currently: zrepl to my personal infra and hexa's, but that's obviously not a good long term

15 - solution.

16 - - Used to have backups to graham's rsync.net account, but that's broken since mid-Jan.

14 + - Currently: zrepl to my personal infra and hexa's, but that's obviously not a

15 + good long term solution.

16 + - Used to have backups to graham's rsync.net account, but that's broken since

17 + mid-Jan.

18 - [raito] Have you ever tried pg_dump's optimized dump format?

19 - [delroth] Is it fast enough to do a daily dump?

20 - [raito] unsure, but there are ways to do incremental backups:

20 - - pg_basebackup + pg_dump compressed format

21 + - pg_basebackup + pg_dump compressed format

23 ## [hexa] Migration of Synapse from EMS

25 - Apparently waiting for EMS to sort out removal of PII?

25 -- [raito] As long as there's discussion happening between Graham and EMS we probably don't have to

26 - care about this, the legacy hosting plan is not getting cancelled.

26 +- [raito] As long as there's discussion happening between Graham and EMS we

27 + probably don't have to care about this, the legacy hosting plan is not getting

28 + cancelled.

29 - [raito] If anything goes wrong we'd likely get notified.

31 ## [eelco] Move fastly log aggregator to pluto

33 - This is currently running on Eelco's local machine which is suboptimal.

32 -- Weekly script that takes Fastly logs and loads them into AWS Athena + generates some aggregates.

34 +- Weekly script that takes Fastly logs and loads them into AWS Athena +

35 + generates some aggregates.

36 - https://github.com/NixOS/infra/tree/master/metrics/fastly

37 - We will put that on the new Eris: Pluto

35 -- [eelco] I will need to create an AWS IAM to bestow the adequate permissions to enable the script

36 - to run on Pluto.

37 - - [eelco] I just need read/write access to Athena and some S3 bucket.

38 +- [eelco] I will need to create an AWS IAM to bestow the adequate permissions to

39 + enable the script to run on Pluto.

40 + - [eelco] I just need read/write access to Athena and some S3 bucket.

41 - [delroth] Who is using this data?

39 - - [eelco] You can see on that page that the reporting is generated via this data

42 + - [eelco] You can see on that page that the reporting is generated via this

43 + data

44 - PII data regarding access logs of cache.nixos.org

41 - - [everyone] What kind of policy do we want regarding PII and the non-critical infrastructure?

42 - e.g. new wiki access logs are available to the non-critical infrastructure

43 - - Let's take note of this, think about it for the next weeks

45 + - [everyone] What kind of policy do we want regarding PII and the non-critical

46 + infrastructure? e.g. new wiki access logs are available to the non-critical

47 + infrastructure

48 + - Let's take note of this, think about it for the next weeks

50 ## [delroth, hexa] Machine changes

47 -Our spend on outdated AWS EC2 instances and EBS volumes is too high and we are cutting back on our

48 -use of EC2 and instead renew our infra at Hetzner.

52 +Our spend on outdated AWS EC2 instances and EBS volumes is too high and we are

53 +cutting back on our use of EC2 and instead renew our infra at Hetzner.

55 - Reduce AWS spending

51 - - Started pruning old snapshots and EBS volumes (e.g. nixos-webserver, old nixos versions)

52 - - [eelco] I think it should be fine to delete them. There's a small risk there could be some

53 - historical data, for instance, our subversion repo used to be there as well and the

54 - nix-dev mailing list too. In theory, we have copies of all of that.

55 - - [delroth] I might start an instance and extract the data out there otherwise I will just

56 - delete it.

57 - - [eelco] There was a lot of scratch space for something… I don't remember it.

58 - - [delroth] I think it was bastion and is now paused.

59 - - Bastion is now stopped/paused

60 - - [hexa] Migrated to Eris and now to Pluto

61 - - [hexa] Channel scripts are running way faster

62 - - [raito] :tada:

63 - - Pinged survey.nixos.org owners (@garbas), to get the limesurvey instance migrated to something

64 - more reasonable

65 - - [hexa] $ 150 USD/mo

66 - - [hexa] Proposal: Migrate to Hetzner Cloud for a fraction of the costs

67 - - [delroth] I asked Julien to look into it

68 - - [delroth] In general, it's open to anyone who are looking to do non-critical work

69 - - Archeology machine from the cache team

70 - - [delroth] Jonas, can you look into the cost? And can we make it start on-demand?

71 - - [jonas] asking edef whether they can accomodate these changes]

56 + - Started pruning old snapshots and EBS volumes (e.g. nixos-webserver, old

57 + nixos versions)

58 + - [eelco] I think it should be fine to delete them. There's a small risk

59 + there could be some historical data, for instance, our subversion repo

60 + used to be there as well and the nix-dev mailing list too. In theory, we

61 + have copies of all of that.

62 + - [delroth] I might start an instance and extract the data out there

63 + otherwise I will just delete it.

64 + - [eelco] There was a lot of scratch space for something… I don't remember

65 + it.

66 + - [delroth] I think it was bastion and is now paused.

67 + - Bastion is now stopped/paused

68 + - [hexa] Migrated to Eris and now to Pluto

69 + - [hexa] Channel scripts are running way faster

70 + - [raito] :tada:

71 + - Pinged survey.nixos.org owners (@garbas), to get the limesurvey instance

72 + migrated to something more reasonable

73 + - [hexa] $ 150 USD/mo

74 + - [hexa] Proposal: Migrate to Hetzner Cloud for a fraction of the costs

75 + - [delroth] I asked Julien to look into it

76 + - [delroth] In general, it's open to anyone who are looking to do

77 + non-critical work

78 + - Archeology machine from the cache team

79 + - [delroth] Jonas, can you look into the cost? And can we make it start

80 + on-demand?

81 + - [jonas] asking edef whether they can accomodate these changes]

82 - Hetzner machine renewal

73 - - Phasing out eris.nixos.org (EX41S-SSD, Intel i7-6700, 64GB RAM, 2x 256GB SATA)

74 - - [hexa] Old hardware

75 - - Created and deployed pluto.nixos.org (EX44, Intel i5-13500, 2x512GB NVME)

76 - - [hexa] Slightly cheaper but modern hardware

77 - - [hexa] Everything migrated except for monitoring

78 - - [delroth] Some disentanglement required to migrate monitoring

79 -

80 -There's a potential of around $700/month of savings in all those operations. That is, we're

81 -offsetting our whole current Hetzner spend with those AWS savings.

83 + - Phasing out eris.nixos.org (EX41S-SSD, Intel i7-6700, 64GB RAM, 2x 256GB

84 + SATA)

85 + - [hexa] Old hardware

86 + - Created and deployed pluto.nixos.org (EX44, Intel i5-13500, 2x512GB NVME)

87 + - [hexa] Slightly cheaper but modern hardware

88 + - [hexa] Everything migrated except for monitoring

89 + - [delroth] Some disentanglement required to migrate monitoring

90 +

91 +There's a potential of around $700/month of savings in all those operations.

92 +That is, we're offsetting our whole current Hetzner spend with those AWS

93 +savings.

95 - [delroth] Future savings (more involved):

84 - - [delroth] Two layers of storage for cache.nixos.org: warm paths on Hetzner

85 - - [delroth] It might be easier to do that stuff on NixOS releases S3 bucket (much smaller

86 - bucket) and it's costing ~1000 USD per month in **bandwidth**

96 + - [delroth] Two layers of storage for cache.nixos.org: warm paths on Hetzner

97 + - [delroth] It might be easier to do that stuff on NixOS releases S3 bucket

98 + (much smaller bucket) and it's costing ~1000 USD per month in **bandwidth**

100 ## [julien] Opening non-critical to more members

101

90 -- [Julien] Idea of non-critical infra was to lower the barrier to entry, because people could be

91 - trusted with less risky infra

92 - - [Julien] I would like to post a Discourse post to look for new people who might be interested

93 - to join the team

94 - - [Julien] It seems like we have some issues open for non-critical infra and let people to

95 - tackle them and could constitute a first project

96 - - [delroth]

97 - https://github.com/NixOS/infra/issues?q=is%3Aopen+is%3Aissue+label%3Anon-critical-infra

98 - - [Julien] I think it's a good time to do such a post and reach out

99 - - [Julien] I wanted to know with everyone if it was okay to invite new people

100 - - [delroth/zimbatm] Yes

101 - - [delroth] I think the most important thing is to know who will take care of onboarding and

102 - leading the work

103 - - [Julien] I am ready to handle the onboarding load and the lead, I would prefer to manage

104 - newcomers rather than do all the stuff by myself

102 +- [Julien] Idea of non-critical infra was to lower the barrier to entry, because

103 + people could be trusted with less risky infra

104 + - [Julien] I would like to post a Discourse post to look for new people who

105 + might be interested to join the team

106 + - [Julien] It seems like we have some issues open for non-critical infra and

107 + let people to tackle them and could constitute a first project

108 + - [delroth]

109 + https://github.com/NixOS/infra/issues?q=is%3Aopen+is%3Aissue+label%3Anon-critical-infra

110 + - [Julien] I think it's a good time to do such a post and reach out

111 + - [Julien] I wanted to know with everyone if it was okay to invite new people

112 + - [delroth/zimbatm] Yes

113 + - [delroth] I think the most important thing is to know who will take care of

114 + onboarding and leading the work

115 + - [Julien] I am ready to handle the onboarding load and the lead, I would

116 + prefer to manage newcomers rather than do all the stuff by myself

117

118 ## [delroth, hexa] Deployment changes

119

108 -We removed nixops and deployment now happens from a `flake.nix`. The plan is to go for colmena

109 -eventually.

120 +We removed nixops and deployment now happens from a `flake.nix`. The plan is to

121 +go for colmena eventually.

122

111 -- Deployment via `nixos-rebuild --flake .#<host> --target-host root@<host>.nixos.org

123 +- Deployment via

124 + `nixos-rebuild --flake .#<host> --target-host root@<host>.nixos.org

125 --use-substitutes switch`

113 -- NixOps generated configuration was imported and is being migrated, for example we:

114 - - started using agenix for secrets management and imported existing secrets

115 - - and migrated Network configuration to systemd-networkd/resolved

116 -

126 +- NixOps generated configuration was imported and is being migrated, for example

127 + we:

128 + - started using agenix for secrets management and imported existing secrets

129 + - and migrated Network configuration to systemd-networkd/resolved

130

131 ## [delroth, hexa] Infra Changelog

132

@@ -121,27 +134,32 @@ eventually.

134 - Migrated haumea's database to PostgreSQL 16

135 - Align timezone across machines

136 - Fix backup of haumea's database

124 - - zrepl to delroth and hexa

125 - - rsync.net stopped working due to zrepl API version mismatch

137 + - zrepl to delroth and hexa

138 + - rsync.net stopped working due to zrepl API version mismatch

139 - Enabled trimming and scrubbing on all ZFS pools

140

141 - Fix the fastly-exporter deployment

129 - - Migrated to nixpkgs module, which [required its own

130 - fixes](https://github.com/NixOS/nixpkgs/pull/287348)

131 - - Generated a new API token, the old one was invalid

132 - - 📊 [Dashboard](https://monitoring.nixos.org/grafana/d/SHjM6e-ik/fastly?orgId=1)

133 -- Fixed [race condition and world-writable state

134 - file](https://github.com/packethost/prometheus-packet-sd/issues/15) upstream in packet-sd

142 + - Migrated to nixpkgs module, which

143 + [required its own fixes](https://github.com/NixOS/nixpkgs/pull/287348)

144 + - Generated a new API token, the old one was invalid

145 + - 📊

146 + [Dashboard](https://monitoring.nixos.org/grafana/d/SHjM6e-ik/fastly?orgId=1)

147 +- Fixed

148 + [race condition and world-writable state

149 + file](https://github.com/packethost/prometheus-packet-sd/issues/15) upstream

150 + in packet-sd

151 - Added alerting for

136 - - Failed systemd units

137 - - [Domain expiry](https://github.com/NixOS/infra/pull/249) within the next 30 days

152 + - Failed systemd units

153 + - [Domain expiry](https://github.com/NixOS/infra/pull/249) within the next 30

154 + days

155 - Lazy loading of eval errors on hydra (Patch by @ajs124)

139 - - Reduces page sizes on the common jobsets/evals by 15-20MB to a few kBs

140 - - More work needed, because error logs are still being fetched from the DB, just not rendered

156 + - Reduces page sizes on the common jobsets/evals by 15-20MB to a few kBs

157 + - More work needed, because error logs are still being fetched from the DB,

158 + just not rendered

159 - Services migrated to pluto.nixos.org

142 - - channel-scripts/hydra-mirror

143 - - netboot

144 - - rfc39

160 + - channel-scripts/hydra-mirror

161 + - netboot

162 + - rfc39

163 - Removed and refactored legacy code, e.g.

146 - - hydra-provisioner

147 - - delft/network.nix

164 + - hydra-provisioner

165 + - delft/network.nix

docs/meeting-notes/2024-03-07.md

+84 -61

@@ -1,71 +1,86 @@

1 # 2024-03-07

3 -Attendees: hexa, vcunat, zimbatm (Jonas), Linus, Julien, Raito/Ryan, Jade (most of the time)

3 +Attendees: hexa, vcunat, zimbatm (Jonas), Linus, Julien, Raito/Ryan, Jade (most

4 +of the time)

6 ## [hexa] arm64 hetzner machine config

8 - Dump it into a new directory in the infra repo, allow infra-build to deploy

8 - - vcunat: There's an issue containing the bits of the configuration

9 - - vcunat: I assumed we wanted to migrate it directly to a new deployment system

10 - - hexa: delroth wanted to script out iPXE but this has not panned out yet, we discovered we had DHCP available, which is promising

9 + - vcunat: There's an issue containing the bits of the configuration

10 + - vcunat: I assumed we wanted to migrate it directly to a new deployment

11 + system

12 + - hexa: delroth wanted to script out iPXE but this has not panned out yet, we

13 + discovered we had DHCP available, which is promising

15 ## [zimbatm] Round table

16 +

17 What is on everyone's mind? What are your plans?

19 - Linus:

16 - - Happy to help out with stuff, pairing on with anything

17 - - zimbatm: Do you think we should do a better presentation?

18 - - linus: I think that'd be good

20 + - Happy to help out with stuff, pairing on with anything

21 + - zimbatm: Do you think we should do a better presentation?

22 + - linus: I think that'd be good

23 - hexa:

20 - - Looking at iPXE, hold us back the most right now

21 - - will coordinate with delroth, if he has already anything

22 - - Open to discuss the Ceph scenario

23 - - A lot of discussions ongoing with the self-hosted binary cache, that's good

24 - - We are running into questions that cannot be answered by anyone

25 - - What should be the availability?

26 - - What should be the durability?

27 - - Discussion running in circles right now

28 - - Form a tightr discussion group

29 - - So that you can identify the main points

30 - - And address them

31 - - And not run into circles

24 + - Looking at iPXE, hold us back the most right now

25 + - will coordinate with delroth, if he has already anything

26 + - Open to discuss the Ceph scenario

27 + - A lot of discussions ongoing with the self-hosted binary cache, that's

28 + good

29 + - We are running into questions that cannot be answered by anyone

30 + - What should be the availability?

31 + - What should be the durability?

32 + - Discussion running in circles right now

33 + - Form a tightr discussion group

34 + - So that you can identify the main points

35 + - And address them

36 + - And not run into circles

37 - vcunat:

33 - - Continuously busy with staging iterations

34 - - Unblocking difficult to access machines, e.g. aarch64 machine

35 - - There's actually more of my machines in the infra and that also requires update

36 - - Small benchmarking machine that makes sense:

37 - - t2a

38 - - The point is to have consistent benchmarking data

39 - - Linus: we definitely don't have cloud VMs for benchmarking, we probably want dedicated hardware

40 - - zimbatm: could you potentially create a ticket to make an inventory of your machines?

41 - - vcunat: there's two machines: t2a and t4b only really

38 + - Continuously busy with staging iterations

39 + - Unblocking difficult to access machines, e.g. aarch64 machine

40 + - There's actually more of my machines in the infra and that also requires

41 + update

42 + - Small benchmarking machine that makes sense:

43 + - t2a

44 + - The point is to have consistent benchmarking data

45 + - Linus: we definitely don't have cloud VMs for benchmarking, we probably

46 + want dedicated hardware

47 + - zimbatm: could you potentially create a ticket to make an inventory of your

48 + machines?

49 + - vcunat: there's two machines: t2a and t4b only really

50 - Julien:

43 - - *Short-term*: I would like to onboard more folks on non-critical infrastructure

44 - - I would like to give them tasks to do end to end

45 - - Difficult to do with the current list of tasks atm

46 - - The wiki is also something I also want to get out ASAP

47 - - The technical issues are basically non-existent, just a little bit more work to do

48 - - Then announcements, onboard people to do editorial work, and that's it

49 - - We are near ready to launch

50 - - zimbatm: Bitwarden

51 - - Julien: we need to move the data from old to new and inform the change to the users

52 - - zimbatm: OK, we need to organize that migration

53 - - Julien: we can discuss this async

54 - - Interested also in cache self-hosting discussions

55 - - We have momentum and it'd be nice to have some sort of stance from infra people

56 - - Addressing the recent unrest regarding the public stance of infra on self hosting

57 - - zimbatm: we should/could do a proof of concept so we can get a feeling about how easy is it to operate

51 + - _Short-term_: I would like to onboard more folks on non-critical

52 + infrastructure

53 + - I would like to give them tasks to do end to end

54 + - Difficult to do with the current list of tasks atm

55 + - The wiki is also something I also want to get out ASAP

56 + - The technical issues are basically non-existent, just a little bit more

57 + work to do

58 + - Then announcements, onboard people to do editorial work, and that's it

59 + - We are near ready to launch

60 + - zimbatm: Bitwarden

61 + - Julien: we need to move the data from old to new and inform the change to

62 + the users

63 + - zimbatm: OK, we need to organize that migration

64 + - Julien: we can discuss this async

65 + - Interested also in cache self-hosting discussions

66 + - We have momentum and it'd be nice to have some sort of stance from infra

67 + people

68 + - Addressing the recent unrest regarding the public stance of infra on self

69 + hosting

70 + - zimbatm: we should/could do a proof of concept so we can get a feeling

71 + about how easy is it to operate

72 - Ryan:

73 - Recommend https://github.com/zhaofengli/colmena/pull/198

61 -Things to pick up for infra: https://github.com/NixOS/infra/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Anew-service

75 +Things to pick up for infra:

76 +https://github.com/NixOS/infra/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Anew-service

78 ## [hexa] darwin access

80 - hexa: We have an inventory problem

66 - - What machines exist? What machines should we be able to access?

67 - - Important so we can delegate access and unblock work

68 - -

81 + - What machines exist? What machines should we be able to access?

82 + - Important so we can delegate access and unblock work

83 + -

85 - Braindump

86 - Apple M1 at Hetzner (hydra)

@@ -75,28 +90,36 @@ Things to pick up for infra: https://github.com/NixOS/infra/issues?q=is%3Aissue+

91 ## [hexa] ofBorg access

78 -- hexa: we have some folks who want to work on OfBorg but cannot do because they are not empowered on to do so

79 - - it is also go via buildkite management mechanism from Graham

93 +- hexa: we have some folks who want to work on OfBorg but cannot do because they

94 + are not empowered on to do so

95 + - it is also go via buildkite management mechanism from Graham

97 ## [raito] aarch64.nixos.community management

99 - https://github.com/NixOS/aarch64-build-box/

84 - - managed by community or infra?

85 - - zimbatm: it used to be in the nix-community infra, but because the nix-community does not have access to the Packet account

86 - - hexa: in the past, the worst we had is to debug the kernel issues, which is difficult w/o packet access

87 - - utilized by ofBorg, too, not a problem because we don't need to trust its build results

88 - - zimbatm: will talk with zowoq, who manages the nix-community day-to-day operation

100 + - managed by community or infra?

101 + - zimbatm: it used to be in the nix-community infra, but because the

102 + nix-community does not have access to the Packet account

103 + - hexa: in the past, the worst we had is to debug the kernel issues, which is

104 + difficult w/o packet access

105 + - utilized by ofBorg, too, not a problem because we don't need to trust its

106 + build results

107 + - zimbatm: will talk with zowoq, who manages the nix-community day-to-day

108 + operation

109

110 ## Changelog

111 +

112 - Cancelled the contract for `eris.nixos.org` (ends after 2024-02-28)

92 - - All services have been migrated to pluto.nixos.org

113 + - All services have been migrated to pluto.nixos.org

114 - Set up backups for Prometheus, Grafana, VictoriaMetrics

115 - The primary hostnames for Prometheus and Grafana have changed

95 - - https://prometheus.nixos.org

96 - - https://grafana.nixos.org

97 - - Redirects for the old hostname/path are in place

116 + - https://prometheus.nixos.org

117 + - https://grafana.nixos.org

118 + - Redirects for the old hostname/path are in place

119 - Hydra changes

99 - - Increase pipe size to improve queue-runner performance

100 - - Increased retention interval of Prometheus to two years so we have more history to evaluate these changes

101 -- Builders have received the fix for https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37

120 + - Increase pipe size to improve queue-runner performance

121 + - Increased retention interval of Prometheus to two years so we have more

122 + history to evaluate these changes

123 +- Builders have received the fix for

124 + https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37

125 - GitHub App for wiki.nixos.org so users can log in.

docs/meeting-notes/2024-03-21.md

+45 -35

@@ -3,46 +3,56 @@

3 Attendees: hexa, vcunat, Linus, zimbatm, Eelco, Janik, raitobezariusm, Alex

5 ## Round table

6 +

7 - zimbatm

7 - - had no spoons to think about cache

8 + - had no spoons to think about cache

9 - vcunat

9 - - expensive nixos tests that could be improved

10 - - noticed `nixos-disk-image.drv` steps taking a long time in send/receive phase

10 + - expensive nixos tests that could be improved

11 + - noticed `nixos-disk-image.drv` steps taking a long time in send/receive

12 + phase

13 - hexa

12 - - unhappy with the board decision to let Anduril sponsor, like delroth. At an impass. We need to find a way to work on this together.

13 - - not sure if delroth is ultimately out.

14 - - don't want to burn out if delroth is gone.

15 - - I also don't want to invest time, when the org agrees to military sponsorship.

16 - - Next step: get to the policy, connect with delroth to see if we can keep working on it together or not.

14 + - unhappy with the board decision to let Anduril sponsor, like delroth. At an

15 + impass. We need to find a way to work on this together.

16 + - not sure if delroth is ultimately out.

17 + - don't want to burn out if delroth is gone.

18 + - I also don't want to invest time, when the org agrees to military

19 + sponsorship.

20 + - Next step: get to the policy, connect with delroth to see if we can keep

21 + working on it together or not.

22 - Janik

18 - - opened issue after the open board calls about meeting infrastructure. https://github.com/NixOS/infra/issues/401

19 - - PR with Jitsi probably soon.

20 - - Do we have a database for the pads?

21 - - hexa: should be colocated with the machine.

22 - - Jonas: do we have hardware for this?

23 - - hexa: we can try it on caliban. If it grows too big we can move it.

23 + - opened issue after the open board calls about meeting infrastructure.

24 + https://github.com/NixOS/infra/issues/401

25 + - PR with Jitsi probably soon.

26 + - Do we have a database for the pads?

27 + - hexa: should be colocated with the machine.

28 + - Jonas: do we have hardware for this?

29 + - hexa: we can try it on caliban. If it grows too big we can move it.

30 - Linus

25 - - happy to review what Janik is doing.

26 - - happy to pair with anyone

31 + - happy to review what Janik is doing.

32 + - happy to pair with anyone

33 - Eelco

28 - - what the plan with the self-hosting?

29 - - still in discussion, we intend to a do some exploration with Ceph

30 - - we still need to find a way to pay for the cache.

34 + - what the plan with the self-hosting?

35 + - still in discussion, we intend to a do some exploration with Ceph

36 + - we still need to find a way to pay for the cache.

37 - Alex Ou

32 - - NixCon NA attendee.

33 - - Interested in infrastructure, their main use of NixOS, managing bare-metal fleet of servers

38 + - NixCon NA attendee.

39 + - Interested in infrastructure, their main use of NixOS, managing bare-metal

40 + fleet of servers

41 - Raitobezarius

35 - - Concerned with the state of the infra, due to delroth ragequitting.

36 - - Tigris Data: CDN+S3 built on top of fly.io that migth be interested in sponsoring us.

37 - - Meeting with PCH.org: have hundreds of datacenters, they can offer everything in terms of storage infra.

38 - - Lots of POPs

39 - - Storage

40 - - ...

41 - - Proposition in progress.

42 - - Would like to update the set of people in the infra core for inactive people, in order to be able to reason on who has access, so we can reason about trust.

43 - - Eelco?

44 - - Graham?

45 - - Amine?

46 - - Proposal: remove access, and restore if needed

47 - - Eelco agreed. (Actually Eelco needs to reconsider.)

48 -

42 + - Concerned with the state of the infra, due to delroth ragequitting.

43 + - Tigris Data: CDN+S3 built on top of fly.io that migth be interested in

44 + sponsoring us.

45 + - Meeting with PCH.org: have hundreds of datacenters, they can offer

46 + everything in terms of storage infra.

47 + - Lots of POPs

48 + - Storage

49 + - ...

50 + - Proposition in progress.

51 + - Would like to update the set of people in the infra core for inactive

52 + people, in order to be able to reason on who has access, so we can reason

53 + about trust.

54 + - Eelco?

55 + - Graham?

56 + - Amine?

57 + - Proposal: remove access, and restore if needed

58 + - Eelco agreed. (Actually Eelco needs to reconsider.)

docs/meeting-notes/2024-04-18.md

+57 -44

@@ -5,7 +5,8 @@ Attendees: delroth, Janik, dgrig, vcunat, raitobezarius, hexa, Linus, Weija

5 ## Topics

7 - [delroth] Bringing up the topic of Keycloak / Kanidm again

8 - - We'll probably want it for Jitsi? I'd also love to drop user management stuff from Hydra.

8 + - We'll probably want it for Jitsi? I'd also love to drop user management

9 + stuff from Hydra.

10 - Other use cases:

11 - Wiki? (I'm guessing mediawiki can SAML)

12 - Pads? (for meeting notes that we'd rather not have vandalized)

@@ -14,16 +15,25 @@ Attendees: delroth, Janik, dgrig, vcunat, raitobezarius, hexa, Linus, Weija

15 - [hexa] Requirements:

16 - GitHub login, and being able to read organization membership info

17 - Maybe Dex can do what we want as well? Proxy to backend apps

17 - - @raitobezarius in chat: [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/github/) as well exist

18 - - @raitobezarius in chat: [SATOSA Proxy](https://github.com/IdentityPython/SATOSA) can be used to do SAML2<->Social Login

18 + - @raitobezarius in chat:

19 + [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/github/)

20 + as well exist

21 + - @raitobezarius in chat:

22 + [SATOSA Proxy](https://github.com/IdentityPython/SATOSA) can be used to

23 + do SAML2<->Social Login

25 - [delroth] releases.nixos.org S3 costs

21 - - tl;dr discovered last week that the bandwidth costs rose significantly for no known reason

22 - - Shape of the growth looks organic but there shouldn't really be anything causing it.

26 + - tl;dr discovered last week that the bandwidth costs rose significantly for

27 + no known reason

28 + - Shape of the growth looks organic but there shouldn't really be anything

29 + causing it.

30 - Fastly logs analysis showed nothing interesting.

24 - - Some access is blocked by [eelco not sharing credentials](https://github.com/NixOS/infra/pull/388#discussion_r1545856527)

31 + - Some access is blocked by

32 + [eelco not sharing credentials](https://github.com/NixOS/infra/pull/388#discussion_r1545856527)

33 - Enabled S3 logging, haven't analyzed yet.

26 - - Cost Explorer might be indicating that this isn't actually releases.nixos.org but something else in eu-west-1 also using S3? But then what? (or is Cost Explorer broken? wouldn't be too surprising)

34 + - Cost Explorer might be indicating that this isn't actually

35 + releases.nixos.org but something else in eu-west-1 also using S3? But then

36 + what? (or is Cost Explorer broken? wouldn't be too surprising)

38 - [Janik] Jitsi on non-critical-infra

39 - nixpkgs+infra PRs were reviewed

@@ -32,47 +42,50 @@ Attendees: delroth, Janik, dgrig, vcunat, raitobezarius, hexa, Linus, Weija

43 - [delroth] What do we still not have access to?

36 - | --- | --- | --- | --- | --- | --- | --- |

49 - | 208.83.1.145 | aarch64-darwin | Macstadium | OfBorg builder | want | \- | |

50 - | 208.83.1.173 | x86_64-darwin | Macstadium | OfBorg builder | want | \- | |

51 - | 208.83.1.175 | x86_64-darwin | Macstadium | OfBorg builder | want | \- | |

52 - | 208.83.1.181 | aarch64-darwin | Macstadium | OfBorg builder | want | \- | |

53 - | 208.83.1.186 | x86_64-darwin | Macstadium | OfBorg builder | want | \- | |

55 - | netboot-foundation | x86_64-linux | Equinix | ? | \- | \- | on demand |

46 + | ------------------------------ | -------------- | ---------- | --------------------------- | ---------------------- | ---------------- | ----------------------------------------- |

59 + | 208.83.1.145 | aarch64-darwin | Macstadium | OfBorg builder | want | \- | |

60 + | 208.83.1.173 | x86_64-darwin | Macstadium | OfBorg builder | want | \- | |

61 + | 208.83.1.175 | x86_64-darwin | Macstadium | OfBorg builder | want | \- | |

62 + | 208.83.1.181 | aarch64-darwin | Macstadium | OfBorg builder | want | \- | |

63 + | 208.83.1.186 | x86_64-darwin | Macstadium | OfBorg builder | want | \- | |

65 + | netboot-foundation | x86_64-linux | Equinix | ? | \- | \- | on demand |

74 ## Changelog:

76 - Removed unused apps on the infra repo

67 - - Slack (unused)

77 + - Slack (unused)

78 - Removed apps from release-wiki repo

69 - - HackMD (unused)

79 + - HackMD (unused)

80 - Removed unused apps on the org level

71 - - Bors (discontinued)

72 - - Marvin-MK2 (discontinued)

73 - - Travis-CI (unused)

81 + - Bors (discontinued)

82 + - Marvin-MK2 (discontinued)

83 + - Travis-CI (unused)

84 - Hydra web UI is fast^W not as slow now (+ other improvements)

75 - - https://github.com/NixOS/hydra/commit/6189ba9c5e5308e17a7d1fb7f38443272a70f072

76 - - Queue runner CPU-heavy operations throttling: https://github.com/NixOS/hydra/commit/a51bd392a22fba5b0a0d90e2204a608b78c37ce1

77 -- Fastly shielding location fixed for releases.nixos.org and tarballs.nixos.org (used to go transatlantic for no good reason)

78 -- http:// redirects to https:// for all our S3 buckets except cache.nixos.org (broke nix-index, temporarily reverted)

85 + - https://github.com/NixOS/hydra/commit/6189ba9c5e5308e17a7d1fb7f38443272a70f072

86 + - Queue runner CPU-heavy operations throttling:

87 + https://github.com/NixOS/hydra/commit/a51bd392a22fba5b0a0d90e2204a608b78c37ce1

88 +- Fastly shielding location fixed for releases.nixos.org and tarballs.nixos.org

89 + (used to go transatlantic for no good reason)

90 +- http:// redirects to https:// for all our S3 buckets except cache.nixos.org

91 + (broke nix-index, temporarily reverted)

docs/meeting-notes/2024-05-30.md

+32 -30

@@ -5,46 +5,48 @@ Attendees: hexa, vcunat, zimbatm, kenji, sterni

5 ## Round table

7 - [hexa]

8 - - Updating Hydra to Nix 2.20

9 - - Ran into (known) regression

10 - - https://github.com/NixOS/nix/issues/9961

11 - - vcunat rolled us back to the previous config

12 - - TODO: needs to persist rollback in git

13 - - nixpkgs is stuck on 2.18

14 - - next step: wait on the next stable Nix release (in nixpkgs)

15 - - Did a round of rotating shared passwords: Hetzner, Netlify (setup 2FA), ...

8 + - Updating Hydra to Nix 2.20

9 + - Ran into (known) regression

10 + - https://github.com/NixOS/nix/issues/9961

11 + - vcunat rolled us back to the previous config

12 + - TODO: needs to persist rollback in git

13 + - nixpkgs is stuck on 2.18

14 + - next step: wait on the next stable Nix release (in nixpkgs)

15 + - Did a round of rotating shared passwords: Hetzner, Netlify (setup 2FA), ...

16 - [vcunat]

17 - - Not anything else significant

17 + - Not anything else significant

18 - [kenji]

19 - - Curious visitor

19 + - Curious visitor

20 - [sterni]

21 - - Nothing in particular

21 + - Nothing in particular

23 ## Topics

25 - [hexa] Vaultwarden mail delivery

26 - - prevents onboarding of new people

27 - - https://github.com/NixOS/infra/issues/430

28 - - solution: https://github.com/NixOS/nixos-wiki-infra/blob/main/modules/postfix.nix

29 - - talking to Julian if he can take it, with fallback to hexa

26 + - prevents onboarding of new people

27 + - https://github.com/NixOS/infra/issues/430

28 + - solution:

29 + https://github.com/NixOS/nixos-wiki-infra/blob/main/modules/postfix.nix

30 + - talking to Julian if he can take it, with fallback to hexa

32 - Netlify

32 - - Need to talk to Marketing if GitHub pages would be sufficient

33 - - Netlify provides preview environments

34 - - Annoying because

35 - - it's expensive,

36 - - DNS is crap,

37 - - cost is per-user

38 - - so we have to share a password.

33 + - Need to talk to Marketing if GitHub pages would be sufficient

34 + - Netlify provides preview environments

35 + - Annoying because

36 + - it's expensive,

37 + - DNS is crap,

38 + - cost is per-user

39 + - so we have to share a password.

41 - [hexa] API modernization in sign-binary-cache script

41 - - https://github.com/NixOS/nixos-channel-scripts/pull/72

42 - - Not used for hydra.nixos.org

43 - - Should close the PR and remove the script to not mislead more people

42 + - https://github.com/NixOS/nixos-channel-scripts/pull/72

43 + - Not used for hydra.nixos.org

44 + - Should close the PR and remove the script to not mislead more people

46 - [zimbatm] Wants to transition out of the team

46 - - Talked with hexa previously in private to take over team lead

47 - - The person doing the things should be leading the team

48 - - Transition out over the next month or so

49 - - Maybe focus for the next month could be on making contributing to the infra repo more comfortable, needs more people who contribute to infra feel welcome

50 -

47 + - Talked with hexa previously in private to take over team lead

48 + - The person doing the things should be leading the team

49 + - Transition out over the next month or so

50 + - Maybe focus for the next month could be on making contributing to the infra

51 + repo more comfortable, needs more people who contribute to infra feel

52 + welcome

docs/meeting-notes/2024-06-13.md

+51 -38

@@ -5,68 +5,81 @@ Attendees: hexa, vcunat, Julien (partially), Eelco

5 ## Round table

7 - Julien

8 - - currently otherwise occupied

9 - - wants to finish the Lime survey migration away from AWS EC2 to non-critical infra

8 + - currently otherwise occupied

9 + - wants to finish the Lime survey migration away from AWS EC2 to non-critical

10 + infra

11 - vcunat

11 - - Full disk on Haumea

12 - - Checking on tarball mirroring service

13 - - wasn't working for the last two weeks, we failed to notice

14 - - issue in nixpkgs caused breakdown

15 - - tending to the script and will merge the fixed version back

12 + - Full disk on Haumea

13 + - Checking on tarball mirroring service

14 + - wasn't working for the last two weeks, we failed to notice

15 + - issue in nixpkgs caused breakdown

16 + - tending to the script and will merge the fixed version back

17 - hexa

17 - - Haumea's backup location

18 - - Super write-intensive

19 - - Return to rsync.net

20 - - tried updating delft/* to 24.05 but hydra wouldn't compile

18 + - Haumea's backup location

19 + - Super write-intensive

20 + - Return to rsync.net

21 + - tried updating delft/* to 24.05 but hydra wouldn't compile

22 - Eelco

22 - - Interested in the cost-increase on the release bucket

23 - - March 6xxx USD

24 - - April 9700 USD

25 - - May 8200 USD

26 - - Still increasing as of June

27 - - Need to move forward with the S3 Bucket (Cache & Releases)

28 - - Move data into Glacier, would be cheaper there, but not accessible from cache.nixos.org anymore

29 - - Moving things out of Glacier expensive, cheaper when we batch requests and request them for the next day or so

30 - - Plan to move to Tigris data, they would give us a discount, and egress is currently free

31 - - Need to get the relevant people in a room to make a final decision

32 - - Eelco

33 - - Edef

34 - - Jonas

35 - - Infra Build (hexa, vcunat)

23 + - Interested in the cost-increase on the release bucket

24 + - March 6xxx USD

25 + - April 9700 USD

26 + - May 8200 USD

27 + - Still increasing as of June

28 + - Need to move forward with the S3 Bucket (Cache & Releases)

29 + - Move data into Glacier, would be cheaper there, but not accessible from

30 + cache.nixos.org anymore

31 + - Moving things out of Glacier expensive, cheaper when we batch requests and

32 + request them for the next day or so

33 + - Plan to move to Tigris data, they would give us a discount, and egress is

34 + currently free

35 + - Need to get the relevant people in a room to make a final decision

36 + - Eelco

37 + - Edef

38 + - Jonas

39 + - Infra Build (hexa, vcunat)

41 ## Action items

42 +

43 - Check Prometheus Alerting Pipeline, no Alerts since May 21

44 - File issue about hydra/nix build failures

45 - Schedule call about S3 bucket decision with Eelco, Jonas, Infra-Build

47 ## Full disk on haumea

43 -- The ZFS pool (1 TB) on Haumea has been running full in the last few days, leading to the PostgreSQL database to be unavailable

48 +

49 +- The ZFS pool (1 TB) on Haumea has been running full in the last few days,

50 + leading to the PostgreSQL database to be unavailable

51 - Multiple options

45 - - Reducing number and frequency of snapshots

46 - - 3x5m, 4x15m, 24x1h, 4x1d, 3x1w

47 - - [vcunat] 5 minutes probably excessive

48 - - Replace haumea with a machine with bigger disks

49 - - AX101 ~100 EUR/Mo

50 - - Long-term maybe prune Hydras database

51 - - or set up a new database and copy only the config over

52 + - Reducing number and frequency of snapshots

53 + - 3x5m, 4x15m, 24x1h, 4x1d, 3x1w

54 + - [vcunat] 5 minutes probably excessive

55 + - Replace haumea with a machine with bigger disks

56 + - AX101 ~100 EUR/Mo

57 + - Long-term maybe prune Hydras database

58 + - or set up a new database and copy only the config over

53 -- Spend some more time debugging the situation, if it doesn't work out go for a bigger machine

60 +- Spend some more time debugging the situation, if it doesn't work out go for a

61 + bigger machine

63 ## Acquire rsync.net account for database backups of haumea

56 -- Previously rsync.net, but Account was paid by Graham. He eventually deleted that account

64 +

65 +- Previously rsync.net, but Account was paid by Graham. He eventually deleted

66 + that account

67 - Currently only backup location is on hexa's NAS at home

68 - Backup size is currently 1.7TiB

69 - At 1.2 Cents per GB/Month that would cost ~24 USD/Month for 2TiB

60 - - https://www.rsync.net/signup/order.html

70 + - https://www.rsync.net/signup/order.html

72 ## E-Mail Alias Management

63 -- Rok would like access, so that he can switch around the alias on the streamyard account that the Marketing team uses

73 +

74 +- Rok would like access, so that he can switch around the alias on the

75 + streamyard account that the Marketing team uses

76 - Resource currently managed by Infra-Build

77 - Not enough opinions, discuss in internal infra room instead

79 ### Changelog

80 +

81 - Non-Critical-Infra updated to NixOS 24.05

69 - - migrated to systemd initrd

82 + - migrated to systemd initrd

83 - Local Postfix setup for mail delivery from vault.nixos.org

84 - Owncast instance at live.nixos.org was set up

85 - Synapse Reverse-Proxying uses Unix Domain Sockets now

docs/meeting-notes/2024-06-27.md

+47 -33

@@ -5,43 +5,57 @@ Attendees: edef, hexa, vcunat, zimbatm

5 ## Round table

7 - hexa

8 - - Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting Indices (https://github.com/NixOS/infra/issues/446)

8 + - Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting

9 + Indices (https://github.com/NixOS/infra/issues/446)

10 - - Actionables:

11 - 1. Setup rsync.net account, so we can have a proper backup, and help hexa's pipe

12 - 2. Try lighter compression with lz4 because we are seeing CPU load bottlenecking

13 - 3. https://github.com/NixOS/infra/pull/447

14 - - Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade path not clear because of incompatible DB versions. Might need a fresh instance after talking to the marketing team.

11 + - Actionables:

12 + 1. Setup rsync.net account, so we can have a proper backup, and help hexa's

13 + pipe

14 + 2. Try lighter compression with lz4 because we are seeing CPU load

15 + bottlenecking

16 + 3. https://github.com/NixOS/infra/pull/447

17 + - Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade

18 + path not clear because of incompatible DB versions. Might need a fresh

19 + instance after talking to the marketing team.

21 - vcunat:

17 - - Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup target

18 - - DB crashed due to full disk and would stop Hydra from working

22 + - Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup

23 + target

24 + - DB crashed due to full disk and would stop Hydra from working

26 - edef:

21 - - Discussed with tomberek and jonas with getting the Glacier copy started. For only large objects to keep it simple.

22 - - The release bucket traffic has grown again?

23 - - edef: it doesn't seem that sizable based on the graphs I am watching

24 - - hexa: did you see the chart Eelco posted? they looked worrying

25 - - edef: to the fastly endpoint

26 - - hexa: AWS

27 - - edef: (looking the AWS Price explorer) looks like 1000 USD/month (30 USD/day), not exploded

28 - - 2000/2010 style infra team

29 - - We get this software thrown over and shall run it

30 - - How can Hydra be made future-proof?

31 - - Who maintains Hydra? Who makes sure the software works for the infra stack we can provide?

32 - - hexa: Only Ericson updates Hydra to new Nix versions, probably for CA derivations, not much else is happening

33 - - vcunat: Scale has increased much over the years since Hydra was written, and it hasn't kept up

34 - - edef: too few people to commit and cover stuff

35 - - biggest issues:

36 - - queue-runner cannot compute runnables faster than they are getting consumed

37 - - hydra kept busy with expensive xz compression of all results it gets

27 + - Discussed with tomberek and jonas with getting the Glacier copy started. For

28 + only large objects to keep it simple.

29 + - The release bucket traffic has grown again?

30 + - edef: it doesn't seem that sizable based on the graphs I am watching

31 + - hexa: did you see the chart Eelco posted? they looked worrying

32 + - edef: to the fastly endpoint

33 + - hexa: AWS

34 + - edef: (looking the AWS Price explorer) looks like 1000 USD/month (30

35 + USD/day), not exploded

36 + - 2000/2010 style infra team

37 + - We get this software thrown over and shall run it

38 + - How can Hydra be made future-proof?

39 + - Who maintains Hydra? Who makes sure the software works for the infra stack

40 + we can provide?

41 + - hexa: Only Ericson updates Hydra to new Nix versions, probably for CA

42 + derivations, not much else is happening

43 + - vcunat: Scale has increased much over the years since Hydra was written,

44 + and it hasn't kept up

45 + - edef: too few people to commit and cover stuff

46 + - biggest issues:

47 + - queue-runner cannot compute runnables faster than they are getting

48 + consumed

49 + - hydra kept busy with expensive xz compression of all results it gets

51 - jonas:

40 - - requester pay on the release S3 bucket?

41 - - last rollout resulted in 404 (silent 403s)

42 - - we use the same code as for the cache

43 - - edef: I tried the fastly code for the cache bucket. Tried it on a separate deployment. It doesn't appear to experience the same issues. Doesn't require a privileged token. Not sure how to further debug that.

44 - - could talk about tigris data

45 - - edef: let's get stuff in there

46 - - edef: need to talk to AWS for free egress

47 - - jonas: just the release bucket for now, because we have issues with it

52 + - requester pay on the release S3 bucket?

53 + - last rollout resulted in 404 (silent 403s)

54 + - we use the same code as for the cache

55 + - edef: I tried the fastly code for the cache bucket. Tried it on a separate

56 + deployment. It doesn't appear to experience the same issues. Doesn't

57 + require a privileged token. Not sure how to further debug that.

58 + - could talk about tigris data

59 + - edef: let's get stuff in there

60 + - edef: need to talk to AWS for free egress

61 + - jonas: just the release bucket for now, because we have issues with it

docs/meeting-notes/2024-11-14.md

+160 -120

@@ -1,75 +1,96 @@

1 # 2024-11-14

3 -Attendees: jkarni, zimbatm, mic92, infinisil, kenji, drig/erethon, arian, sam , hexa, jeremy, jeff

3 +Attendees: jkarni, zimbatm, mic92, infinisil, kenji, drig/erethon, arian, sam ,

4 +hexa, jeremy, jeff

6 ## Round Table

8 ### Ofborg

9 -- Mic92: POC to evaluate nixpkgs on GitHub Actions. Results looked promising. nixpkgs-review would run in 5 minutes. the ofborg-eval was heavily swapping and taking 15min.

10 -- Infinisil: people might have to enable GHA in their fork, which is disabled by default

11 - - Mic92: I didn't see this behaviour?

12 - - Kenji: I think Github changed some defaults

13 - - ref: https://github.blog/changelog/2024-11-05-notice-of-breaking-changes-for-github-actions/#changes-to-workflow-validation-for-pull-requests-originating-from-forked-repositories

14 - - Mic92: I think this is true for periodic

15 - - Infinisil: trying now for a new user

16 -- Mic92: if we want to pursue GHA, we would have to evaluate nixpkgs twice because we need to get the store paths for master, and the changes of the PR, and then we can compute all packages that have been changed, and append that textfile as data. This can then be re-used by nixpkgs-review.

10 +- Mic92: POC to evaluate nixpkgs on GitHub Actions. Results looked promising.

11 + nixpkgs-review would run in 5 minutes. the ofborg-eval was heavily swapping

12 + and taking 15min.

13 +- Infinisil: people might have to enable GHA in their fork, which is disabled by

14 + default

15 + - Mic92: I didn't see this behaviour?

16 + - Kenji: I think Github changed some defaults

17 + - ref:

18 + https://github.blog/changelog/2024-11-05-notice-of-breaking-changes-for-github-actions/#changes-to-workflow-validation-for-pull-requests-originating-from-forked-repositories

19 + - Mic92: I think this is true for periodic

20 + - Infinisil: trying now for a new user

21 +- Mic92: if we want to pursue GHA, we would have to evaluate nixpkgs twice

22 + because we need to get the store paths for master, and the changes of the PR,

23 + and then we can compute all packages that have been changed, and append that

24 + textfile as data. This can then be re-used by nixpkgs-review.

25 - Arian: why are we not using the PR workflow?

18 - - Mic92: concurrency issues (limit of 20 runners per org).

19 - - TODO: check if we hit the limit

26 + - Mic92: concurrency issues (limit of 20 runners per org).

27 + - TODO: check if we hit the limit

28 - hexa: the idea to comment on the PR is to compensate for the visibility issue?

21 - - Mic92: yes. it sucks a bit, but this could be mitigated by a small web service.

22 - - A thin wrapper that receives a webhook, checks back the PR status and translates it as a comment.

23 -- Infinisil: wouldn't it be possible to have a workflow that polls on behalf of the user?

24 - - It's a workflow that tries to find the workflow on the user's push, in their fork.

25 - - It would be triggered every time you synchronize the PR.

26 - - Mic92: can you set this up in a way that the workflow gets triggered once the workflow is finished?

27 - - Infinisil: I think you need to poll for this.

28 - - Mic92: Is it a 1:1 mapping, or 1:N?

29 - - infinisil: Something like

30 - ```yaml

31 - # .github/workflows/query-pr.yml

32 - on: pull_request_target

33 - jobs:

34 - check:

35 - runs-on: ubuntu-latest

36 - steps:

37 - - run: |

38 - gh api /repos/BASE_REPO/commits/GITHUB_SHA/check-runs

39 - ```

40 - - Worry: Offloading OfBorg on to GH could give us trouble, because it might not be insignificant compute.

29 + - Mic92: yes. it sucks a bit, but this could be mitigated by a small web

30 + service.

31 + - A thin wrapper that receives a webhook, checks back the PR status and

32 + translates it as a comment.

33 +- Infinisil: wouldn't it be possible to have a workflow that polls on behalf of

34 + the user?

35 + - It's a workflow that tries to find the workflow on the user's push, in their

36 + fork.

37 + - It would be triggered every time you synchronize the PR.

38 + - Mic92: can you set this up in a way that the workflow gets triggered once

39 + the workflow is finished?

40 + - Infinisil: I think you need to poll for this.

41 + - Mic92: Is it a 1:1 mapping, or 1:N?

42 + - infinisil: Something like

43 + ```yaml

44 + # .github/workflows/query-pr.yml

45 + on: pull_request_target

46 + jobs:

47 + check:

48 + runs-on: ubuntu-latest

49 + steps:

50 + - run: |

51 + gh api /repos/BASE_REPO/commits/GITHUB_SHA/check-runs

52 + ```

53 + - Worry: Offloading OfBorg on to GH could give us trouble, because it might

54 + not be insignificant compute.

55 - Jeremy: is this confined to PRs, or running on all branches?

42 - - Mic92: it would be on push, but checking if the branch is part of the PR.

43 - - Mic92: actually, there might be some synchronicity issue, because the PR happens after the push.

44 - - Mic92: Because of that we might need a webservice that can trigger actions

45 - - Mic92: can we get an event when we open a PR?

56 + - Mic92: it would be on push, but checking if the branch is part of the PR.

57 + - Mic92: actually, there might be some synchronicity issue, because the PR

58 + happens after the push.

59 + - Mic92: Because of that we might need a webservice that can trigger actions

60 + - Mic92: can we get an event when we open a PR?

47 -- Arian: Team plan gives us 60 concurrent actions by the way. (And team plan is free for non-profit orgs)

62 +- Arian: Team plan gives us 60 concurrent actions by the way. (And team plan is

63 + free for non-profit orgs)

65 - Pushes don't have a base branch, need a base branch to compare the out paths

50 - - Mic92: only if not open as a PR

51 - - infinisil: Can pre-compute the out paths on push, cache out paths on Nixpkgs master, then comparing can be done in a PR action fairly easily

66 + - Mic92: only if not open as a PR

67 + - infinisil: Can pre-compute the out paths on push, cache out paths on Nixpkgs

68 + master, then comparing can be done in a PR action fairly easily

70 - Jonas: who is going to make this happen?

54 - - Infinisil, Alex Balsoft, Jeremy after early December, Mic92 can write some scripts and don't want to lead (want to work on the binary cache).

71 + - Infinisil, Alex Balsoft, Jeremy after early December, Mic92 can write some

72 + scripts and don't want to lead (want to work on the binary cache).

74 - Infinisil: not convinced if that's a good idea.

76 - Jonas: What would it take to get to feature parity

59 - - What is the minimal set?

60 - - Mic92: Minimal - Evaluate Prs

61 - - Mic92: 2nd Phase - We can build packages.

62 - - Mic92: Labels for mass-rebuilds

63 - - Silvan: Requests reviews from maintainers (maybe not needed?)

64 - - Silvan: Don't need to build manual with OfBorg anymore (Is already built in ci)

65 - - Silvan: Evaluating without aliases

66 - - Jonas: Discourage IFD's

67 - - Silvan: Maybe it really is good to split this up in two parts:

68 - - Evaluating

69 - - Building

70 -

71 -- Arian: Average job queue time is currently 9s: https://github.com/NixOS/nixpkgs/actions/metrics/performance

72 -- Arian: I would aim for: Lets just try with `pull_request:` and only do the complicated `push:` abuse if that job queue time is gonna go up significantly

77 + - What is the minimal set?

78 + - Mic92: Minimal - Evaluate Prs

79 + - Mic92: 2nd Phase - We can build packages.

80 + - Mic92: Labels for mass-rebuilds

81 + - Silvan: Requests reviews from maintainers (maybe not needed?)

82 + - Silvan: Don't need to build manual with OfBorg anymore (Is already built in

83 + ci)

84 + - Silvan: Evaluating without aliases

85 + - Jonas: Discourage IFD's

86 + - Silvan: Maybe it really is good to split this up in two parts:

87 + - Evaluating

88 + - Building

89 +

90 +- Arian: Average job queue time is currently 9s:

91 + https://github.com/NixOS/nixpkgs/actions/metrics/performance

92 +- Arian: I would aim for: Lets just try with `pull_request:` and only do the

93 + complicated `push:` abuse if that job queue time is gonna go up significantly

94 - Jonas: Looking forward to having eval failures to block merges!

96 - Silvan: Who can review pr's and help out:

@@ -81,84 +102,103 @@ Attendees: jkarni, zimbatm, mic92, infinisil, kenji, drig/erethon, arian, sam ,

102 - Silvan: TODO - Mention this effort on Discourse

103

104 - Mic92: How do we coordinate?

84 - - Main Evalation

85 - - Figure out parts we can parallelize:

86 - - most parts are fairly orthogonal

105 + - Main Evalation

106 + - Figure out parts we can parallelize:

107 + - most parts are fairly orthogonal

108

109 - Silvan: Somebody could lead the Building part:

89 - - Mic92: Find someone who can help out, maybe on discourse?

90 - - Silvan: GH doesn't have all the architectures

91 - - Mic92: Start with the ones we have currently

92 - - Silvan: If we don't need ealuation anymore - this could save a lot of resources, could optimize

93 - - hexa: Yes, but it might not apply on top of staging for example

94 - - Silvan: Yes, staging can probably be ignored

95 - - hexa: Yes, it tries to build against the target branch, led to some problems, for example always trying to build llvm on darwin -> continuuous timeouts

96 - - Mic92: A ton of stuff we could potentially optimize

97 - - Silvan: Empower users to build on more architectures

98 - - Mic92: Convenient to have logs in public

99 - - Mic92: I would like to see a /build command, so that builds can be manually triggered

110 + - Mic92: Find someone who can help out, maybe on discourse?

111 + - Silvan: GH doesn't have all the architectures

112 + - Mic92: Start with the ones we have currently

113 + - Silvan: If we don't need ealuation anymore - this could save a lot of

114 + resources, could optimize

115 + - hexa: Yes, but it might not apply on top of staging for example

116 + - Silvan: Yes, staging can probably be ignored

117 + - hexa: Yes, it tries to build against the target branch, led to some

118 + problems, for example always trying to build llvm on darwin -> continuuous

119 + timeouts

120 + - Mic92: A ton of stuff we could potentially optimize

121 + - Silvan: Empower users to build on more architectures

122 + - Mic92: Convenient to have logs in public

123 + - Mic92: I would like to see a /build command, so that builds can be manually

124 + triggered

125

126 - Silvan: Optimization of the Eval part:

102 - - look at path that actually changed

103 - - mic92: Aware of nix script that gives names of paths that are actually changed?

104 - - Silvan: Yes

105 - - Mic92: If heavy swapping, then it might speed up, else we might see a slow down

127 + - look at path that actually changed

128 + - mic92: Aware of nix script that gives names of paths that are actually

129 + changed?

130 + - Silvan: Yes

131 + - Mic92: If heavy swapping, then it might speed up, else we might see a slow

132 + down

133

134 - Silvan: Where should we report?

108 - - discourse?

109 - - Mic92: Discussion would be nicer on GH, because we can link issues/pr's.

135 + - discourse?

136 + - Mic92: Discussion would be nicer on GH, because we can link issues/pr's.

137

138 ## Topics

139

140 - Transfer of the Macs located at Detsys to Flying Circus

114 - - Scheduled for 2024-11-25

115 - - Currently enrolled into Detsys MDM Account. Can we set something up to migrate that to an infra team account?

116 - - Report back the result to the Mac Mini Logistics room on Matrix

117 - - MDM built into macs, but need to be enrolled into an mdm vendor

118 - - Arian: Don't have to do it, but very convenient

119 - - Mic92: if there are no major problems, should look into MDM as well.

141 + - Scheduled for 2024-11-25

142 + - Currently enrolled into Detsys MDM Account. Can we set something up to

143 + migrate that to an infra team account?

144 + - Report back the result to the Mac Mini Logistics room on Matrix

145 + - MDM built into macs, but need to be enrolled into an mdm vendor

146 + - Arian: Don't have to do it, but very convenient

147 + - Mic92: if there are no major problems, should look into MDM as well.

148 - Oakhost Macs are available

121 - - Need the usual setup

122 - - 3 new machines

123 - - arian can set this up

124 - - initial password

125 - - Mic92: Mac enrollment not very automated yet, last time hexa wrote some stuff down

126 - - Arians keys needs to be added to the repository

127 - - Arian:

128 - - need access to oakhost

129 - - ssh key to infra

149 + - Need the usual setup

150 + - 3 new machines

151 + - arian can set this up

152 + - initial password

153 + - Mic92: Mac enrollment not very automated yet, last time hexa wrote some

154 + stuff down

155 + - Arians keys needs to be added to the repository

156 + - Arian:

157 + - need access to oakhost

158 + - ssh key to infra

159 - Mac Issues:

131 - - hexa: Forking issues on seqoia

132 - - hexa: Running quite well atm

133 - - hexa: Patched out chrooting of nix, applied patches on top of darwin builders

134 - - hexa: Not sure exactly why that works

135 - - hexa: Upstreaming rosetta2-gc to nix-darwin currently

136 - - hexa: darwin 15.1 had issues -> hetzner doesn't roll back (need rescue mode)

137 - - hexa: Rollbacks likely possible with mdm

138 - - arian: We can likely add hetzner darwin machines to be managed by mdm, but not too sure

139 - - arian: Looks into if we can add without physical access, look into what detsys did

160 + - hexa: Forking issues on seqoia

161 + - hexa: Running quite well atm

162 + - hexa: Patched out chrooting of nix, applied patches on top of darwin

163 + builders

164 + - hexa: Not sure exactly why that works

165 + - hexa: Upstreaming rosetta2-gc to nix-darwin currently

166 + - hexa: darwin 15.1 had issues -> hetzner doesn't roll back (need rescue mode)

167 + - hexa: Rollbacks likely possible with mdm

168 + - arian: We can likely add hetzner darwin machines to be managed by mdm, but

169 + not too sure

170 + - arian: Looks into if we can add without physical access, look into what

171 + detsys did

172 - Equinix Metal Exit Plan

141 - - https://md.darmstadt.ccc.de/eqm-exit-plan

142 - - hexa: This is what we had, this is what we need, this is what it is going to cost.

143 - - mic92: Should we check out what we need for the arm64 builders?

144 - - hexa: Basically choice between: 64GB, or 256GB of memory

145 - - hexa: Likely want the bigger memory

146 - - Mic92: Can try the same for arm64, for x86 we can look in to funding.

147 - - Mic92: How long do we need for set up? - a day? Shouldn't take long to set up.

148 - - Mic92: I will set this in motion, unless someone else want's to reach out.

149 - - Mic92: Ok, I will do it.

150 - - hexa: Ideally we don't have 20 small machine, but 5 big machines. Which would be good for maintenance reasons. Because we likely won't get a netboot setup anymore.

173 + - https://md.darmstadt.ccc.de/eqm-exit-plan

174 + - hexa: This is what we had, this is what we need, this is what it is going to

175 + cost.

176 + - mic92: Should we check out what we need for the arm64 builders?

177 + - hexa: Basically choice between: 64GB, or 256GB of memory

178 + - hexa: Likely want the bigger memory

179 + - Mic92: Can try the same for arm64, for x86 we can look in to funding.

180 + - Mic92: How long do we need for set up? - a day? Shouldn't take long to set

181 + up.

182 + - Mic92: I will set this in motion, unless someone else want's to reach out.

183 + - Mic92: Ok, I will do it.

184 + - hexa: Ideally we don't have 20 small machine, but 5 big machines. Which

185 + would be good for maintenance reasons. Because we likely won't get a netboot

186 + setup anymore.

187 - Security Tracker

152 - - dgrig: Jonas gave me access to a Hetzner Cloud project a couple of weeks ago. A VM is up and running, I'm figuring out how to implement this in the same way as nixos-infra.

153 - - dgrig: Do we care about having this in Terraform? I used TF to spin this up, but the state is in my computer currently, do we care to push it to S3?

154 - - drgrig: Can I do `nixos-installer --flake nixos/nixos-infra`?How can I install the nixos-infra

155 - - Mic02: Inputs :"${inputs.nixos-infra}/keys" can convert to string.

156 - - Example:

157 - ```nix

158 - users.users.root.openssh.authorizedKeys.keys = [] ++ (builtins.filter (l: l != [ ]) (builtins.split "\n" (builtins.readFile inputs.phaer-keys)));

159 - ```

160 - - dgrig: Do we care about the Terraform state yet?

161 - - hexa: Do we need the state yet?

162 - - drgrid: It is the tf state

163 - - consensus: We don't care

164 - - Jeremy: Nit - export the keys as a module

188 + - dgrig: Jonas gave me access to a Hetzner Cloud project a couple of weeks

189 + ago. A VM is up and running, I'm figuring out how to implement this in the

190 + same way as nixos-infra.

191 + - dgrig: Do we care about having this in Terraform? I used TF to spin this up,

192 + but the state is in my computer currently, do we care to push it to S3?

193 + - drgrig: Can I do `nixos-installer --flake nixos/nixos-infra`?How can I

194 + install the nixos-infra

195 + - Mic02: Inputs :"${inputs.nixos-infra}/keys" can convert to string.

196 + - Example:

197 + ```nix

198 + users.users.root.openssh.authorizedKeys.keys = [] ++ (builtins.filter (l: l != [ ]) (builtins.split "\n" (builtins.readFile inputs.phaer-keys)));

199 + ```

200 + - dgrig: Do we care about the Terraform state yet?

201 + - hexa: Do we need the state yet?

202 + - drgrid: It is the tf state

203 + - consensus: We don't care

204 + - Jeremy: Nit - export the keys as a module

formatter/flake-module.nix

@@ -15,6 +15,7 @@

15 ];

17 programs.actionlint.enable = true;

18 + programs.deno.enable = true;

19 programs.terraform.enable = true;

20 programs.deadnix.enable = true;

21 programs.nixfmt.enable = true;

hydra-packet-importer/README.md

+2 -2

@@ -1,5 +1,5 @@

1 -Imports builders' and their SSH keys from the Packet API. Requires the

2 -builder run this script at startup:

1 +Imports builders' and their SSH keys from the Packet API. Requires the builder

2 +run this script at startup:

4 ```bash

5 #!/usr/bin/env nix-shell

hydra-packet-importer/config-example.json

+30 -30

@@ -1,34 +1,34 @@

1 {

2 - "token": "You read only API token",

3 - "project_id": "Your project's unique ID",

4 - "mandatory_tags": [ "tags which each", "server", "must have" ],

5 - "skip_tags": [ "tags which will", "exclude a server from", "being imported" ],

6 - "plans": {

7 - "c2.large.arm": {

8 - "user": "root",

9 - "system_types": ["aarch64-linux"],

10 - "ssh_key": "/path/to/hydras-ssh-key",

11 - "max_jobs": 15,

12 - "speed_factor": 1,

13 - "features": ["kvm","nixos-test","big-parallel"],

14 - "mandatory_features": []

15 - },

16 - "c2.medium.x86": {

17 - "user": "root",

18 - "system_types": ["x86_64-linux", "i686-linux"],

19 - "ssh_key": "/var/lib/hydra/queue-runner/.ssh/id_buildfarm_rsa",

20 - "max_jobs": 18,

21 - "speed_factor": 1,

22 - "features": [ "kvm","nixos-test","big-parallel" ]

23 - }

2 + "token": "You read only API token",

3 + "project_id": "Your project's unique ID",

4 + "mandatory_tags": ["tags which each", "server", "must have"],

5 + "skip_tags": ["tags which will", "exclude a server from", "being imported"],

6 + "plans": {

7 + "c2.large.arm": {

8 + "user": "root",

9 + "system_types": ["aarch64-linux"],

10 + "ssh_key": "/path/to/hydras-ssh-key",

11 + "max_jobs": 15,

12 + "speed_factor": 1,

13 + "features": ["kvm", "nixos-test", "big-parallel"],

14 + "mandatory_features": []

15 },

25 - "name_overrides": {

26 - "machines-with-this-name-override-defaults-in-plan": {

27 - "max_jobs": 20,

28 - "system_types": [ "x86_64-linux" ],

29 - "max_jobs": 1,

30 - "speed_factor": 100,

31 - "features": [ "kvm","nixos-test","big-parallel" ]

32 - }

16 + "c2.medium.x86": {

17 + "user": "root",

18 + "system_types": ["x86_64-linux", "i686-linux"],

19 + "ssh_key": "/var/lib/hydra/queue-runner/.ssh/id_buildfarm_rsa",

20 + "max_jobs": 18,

21 + "speed_factor": 1,

22 + "features": ["kvm", "nixos-test", "big-parallel"]

23 }

24 + },

25 + "name_overrides": {

26 + "machines-with-this-name-override-defaults-in-plan": {

27 + "max_jobs": 20,

28 + "system_types": ["x86_64-linux"],

29 + "max_jobs": 1,

30 + "speed_factor": 100,

31 + "features": ["kvm", "nixos-test", "big-parallel"]

32 + }

33 + }

34 }

macs/README.md

+7 -6

@@ -6,15 +6,16 @@ See [inventory](../docs/inventory.md).

7 ### At Graham's place

9 -We have mac-mini's are in [Grahams](https://github.com/grahamc) house,

10 -that only [@cole-h](https://github.com/cole-h) can deploy:

9 +We have mac-mini's are in [Grahams](https://github.com/grahamc) house, that only

10 +[@cole-h](https://github.com/cole-h) can deploy:

12 - becoming-hyena.foundation.detsys.dev

13 - cosmic-stud.foundation.detsys.dev

14 - quality-ram.foundation.detsys.dev

15 - tight-bug.foundation.detsys.dev

17 -- These are getting erased and automatically redeployed from the configuration in this directory.

17 +- These are getting erased and automatically redeployed from the configuration

18 + in this directory.

20 ### Hetzner

@@ -42,14 +43,15 @@ These are maintained by the build infra team.

43 - ~~softwareupdate --install --all --restart~~

44 - Disable auto-updates:

45 - We are currently seeing performance regression in macOS Sequoia.

45 - - So to not have the machines auto-upgrade, we use: `sudo softwareupdate --schedule off`

46 + - So to not have the machines auto-upgrade, we use:

47 + `sudo softwareupdate --schedule off`

48 - Install rosetta2

49 - softwareupdate --install-rosetta2 --agree-to-license

50 - Set up passwordless sudo

51 ```

52 # visudo /etc/sudoers.d/passwordless

53 %admin ALL = NOPASSWD: ALL

52 - ````

54 + ```

55 - Install nix

56 - `sh <(curl -L https://nixos.org/nix/install) --daemon`

57 - Install nix-darwin

@@ -61,4 +63,3 @@ These are maintained by the build infra team.

63 ```

64 darwin-rebuild switch --flake github:nixos/infra#arm64

65 ```

64 -

metrics/fastly/README.md

+54 -58

@@ -2,21 +2,20 @@

3 This flake provides a systemd timer (`./cron.sh`) that every week:

5 -* Ingests raw Fastly logs for

6 - {cache,channels,tarballs,releases}.nixos.org (which are very big)

7 - and aggregates them into a smaller AWS Athena database.

5 +- Ingests raw Fastly logs for {cache,channels,tarballs,releases}.nixos.org

6 + (which are very big) and aggregates them into a smaller AWS Athena database.

8 This is performed by `./ingest-raw-logs.sh`.

11 -* Runs a number of SQL queries against the Athena database and stores

12 - them in S3.

10 +- Runs a number of SQL queries against the Athena database and stores them in

11 + S3.

13 This is performed by `./run-queries.sh`.

15 ## AWS Athena database

18 -The Athena database is stored in the NixOS Foundation AWS account. To

19 -get the schema, run

17 +The Athena database is stored in the NixOS Foundation AWS account. To get the

18 +schema, run

20 ```

21 # aws athena list-table-metadata --region eu-west-1 --catalog-name AwsDataCatalog --database-name default

@@ -24,26 +23,26 @@ get the schema, run

24 It has the following external tables:

27 -* `requests`: An external table. These are the raw fastly logs stored

28 - in s3://fastly-logs-20220622145016462800000001/ as compressed JSON

29 - records. Note that this bucket has a lifecycle rule that moves logs

30 - to Glacier after a few weeks. Logs in Glacier are not processed by

31 - Athena.

26 +- `requests`: An external table. These are the raw fastly logs stored in

27 + s3://fastly-logs-20220622145016462800000001/ as compressed JSON records. Note

28 + that this bucket has a lifecycle rule that moves logs to Glacier after a few

29 + weeks. Logs in Glacier are not processed by Athena.

33 -* `asn_list`: A list of ASNs. This can be updated by running

31 +- `asn_list`: A list of ASNs. This can be updated by running

32 `./update-asn-list.sh`.

36 -* `hosting-asns`: A list of ASNs belonging to hosting/cloud providers.

34 +- `hosting-asns`: A list of ASNs belonging to hosting/cloud providers.

38 -* `all_paths`: The set of all store paths known in the hydra.nixos.org

39 - database. This is used to expand the hash part of `.narinfo` requests

40 - (e.g. `8kbx6s9nn7060zsdms3br0mk7bjrvbij`) to store paths

41 - (e.g. `/nix/store/8kbx6s9nn7060zsdms3br0mk7bjrvbij-coreutils-full-9.0`).

36 +- `all_paths`: The set of all store paths known in the hydra.nixos.org database.

37 + This is used to expand the hash part of `.narinfo` requests (e.g.

38 + `8kbx6s9nn7060zsdms3br0mk7bjrvbij`) to store paths (e.g.

39 + `/nix/store/8kbx6s9nn7060zsdms3br0mk7bjrvbij-coreutils-full-9.0`).

41 FIXME: describe how to update.

45 -* `release_paths`: All the store paths belonging to NixOS evals in

46 - hydra.nixos.org, as `{project, jobset, eval, release_name, build,

43 +- `release_paths`: All the store paths belonging to NixOS evals in

44 + hydra.nixos.org, as

45 + `{project, jobset, eval, release_name, build,

46 output, path}` tuples.

48 FIXME: describe how to update.

@@ -51,70 +50,67 @@ It has the following external tables:

50 The ingestion script populates the following tables stored in

51 s3://nixos-athena/fastly-logs-processed/:

54 -* `urls`: For each host/day/url, the total number of requests, bytes

55 - and elapsed microseconds. This only includes info about successful

56 - (2xx/3xx) requests.

53 +- `urls`: For each host/day/url, the total number of requests, bytes and elapsed

54 + microseconds. This only includes info about successful (2xx/3xx) requests.

58 -* `clients`: For each host/day/ASN/country/region, the total number of

59 - requests, bytes and elapsed microseconds.

56 +- `clients`: For each host/day/ASN/country/region, the total number of requests,

57 + bytes and elapsed microseconds.

61 -* `nix_cache_info`: For each day/ASN/country/region/user-agent, the

62 - number of requests for `nix-cache-info`.

59 +- `nix_cache_info`: For each day/ASN/country/region/user-agent, the number of

60 + requests for `nix-cache-info`.

62 ## Reports

64 Currently the following reports are created every week:

68 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-day.csv

66 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-day.csv

70 - For each day and site, the number of requests and the number of

71 - bytes transferred.

68 + For each day and site, the number of requests and the number of bytes

69 + transferred.

73 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-country.csv

71 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-country.csv

75 - For each country, the number of requests and the number of

76 - bytes transferred.

73 + For each country, the number of requests and the number of bytes transferred.

78 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day.csv

75 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day.csv

77 For each day, the number of requests for

78 https://cache.nixos.org/nix-cache-info.

83 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-not-hosted.csv

80 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-not-hosted.csv

85 - The same, but with requests from "hosting" ASNs (e.g. AWS and

86 - Hetzner) filtered out. Note that Nix caches `nix-cache-info` file

87 - for a week, so the intent of this report is to gauge the number of

88 - active weekly users.

82 + The same, but with requests from "hosting" ASNs (e.g. AWS and Hetzner)

83 + filtered out. Note that Nix caches `nix-cache-info` file for a week, so the

84 + intent of this report is to gauge the number of active weekly users.

90 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-per-ua.csv

86 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-per-ua.csv

92 - For each day and user agent (e.g. `Nix/2.12.0`), the number of

93 - requests for https://cache.nixos.org/nix-cache-info. This is

94 - intended to track the adoption of Nix releases.

88 + For each day and user agent (e.g. `Nix/2.12.0`), the number of requests for

89 + https://cache.nixos.org/nix-cache-info. This is intended to track the adoption

90 + of Nix releases.

96 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/flake-registry-requests-per-day.csv

92 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/flake-registry-requests-per-day.csv

94 For each day, the number of requests for

99 - https://channels.nixos.org/flake-registry.json. This is intended to

100 - track how widely flakes are used.

95 + https://channels.nixos.org/flake-registry.json. This is intended to track how

96 + widely flakes are used.

102 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/top-store-paths.csv

98 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/top-store-paths.csv

104 - For each store path listed in `all_paths`, the number of requests for its `.narinfo` file.

100 + For each store path listed in `all_paths`, the number of requests for its

101 + `.narinfo` file.

102

106 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/narinfo-queries-per-release.csv

103 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/narinfo-queries-per-release.csv

104

108 - For each major NixOS release (e.g. `nixos-22.05`), the number of

109 - requests for `.narinfo` files of store paths that are part of an

110 - eval of that release.

105 + For each major NixOS release (e.g. `nixos-22.05`), the number of requests for

106 + `.narinfo` files of store paths that are part of an eval of that release.

107

112 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-downloads.csv

108 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-downloads.csv

109

114 - For each day, the number of downloads of the Nix installer

115 - (i.e. `https://releases.nixos.org/nix/nix-[^/]+/install`).

110 + For each day, the number of downloads of the Nix installer (i.e.

111 + `https://releases.nixos.org/nix/nix-[^/]+/install`).

112

117 -* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-architectures.csv

113 +- http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-architectures.csv

114

119 - For each architecture (e.g. `x86_64-linux`), the number of downloads

120 - of the Nix binary tarball.

115 + For each architecture (e.g. `x86_64-linux`), the number of downloads of the

116 + Nix binary tarball.

non-critical-infra/.sops.yaml

+9 -9

@@ -1,4 +1,4 @@

1 -keys:

1 +keys:

2 - &hexa age1j3mkgedmeru63vwww6m44zfw09tg8yw6xdzstaq7ejfkvgcau40qwakm8x

3 - &zimbatm age1jrh8yyq3swjru09s75s4mspu0mphh7h6z54z946raa9wx3pcdegq0x8t4h

4 - &caliban age1sv307kkrxwgjah8pjpap5kzl4j2r6fqr3vg234n7m32chlchs9lsey7nlq

@@ -7,14 +7,14 @@ keys:

7 creation_rules:

8 - path_regex: secrets/[^/]+.caliban

9 key_groups:

10 - - age:

11 - - *caliban

12 - - *hexa

13 - - *zimbatm

10 + - age:

11 + - *caliban

12 + - *hexa

13 + - *zimbatm

15 - path_regex: secrets/[^/]+.umbriel

16 key_groups:

17 - - age:

18 - - *umbriel

19 - - *hexa

20 - - *zimbatm

17 + - age:

18 + - *umbriel

19 + - *hexa

20 + - *zimbatm

non-critical-infra/README.md

+15 -9

@@ -1,19 +1,25 @@

1 -Non-critical-infra

2 -================

1 +# Non-critical-infra

4 -This folder of the repository contains all files relative to the non-critical infra team. Machines managed by that specific configuration are distinct from the ones used in the rest of that repository and used to host services useful to the general Nix/NixOS community.

3 +This folder of the repository contains all files relative to the non-critical

4 +infra team. Machines managed by that specific configuration are distinct from

5 +the ones used in the rest of that repository and used to host services useful to

6 +the general Nix/NixOS community.

6 -

7 -## For the users

8 +## For the users

10 ### I would like my project hosted by this infrastructure

10 -Open a PR or an issue, and members of the infra team will tell you if this infrastructure is suitable to the project!

11 +

12 +Open a PR or an issue, and members of the infra team will tell you if this

13 +infrastructure is suitable to the project!

15 ### I would like to join the team

13 -Come and talk to us on matrix: #infra:nixos.org

17 +Come and talk to us on matrix: #infra:nixos.org

19 ## For the contributors

18 -### Secret access

19 -Secret access is on a "need to have" basis. If you think you need access to the secrets, please add your key to the `.sops.yaml` file on a PR and ping people that already have access for them to run the `updatekeys` command.

21 +### Secret access

22 +

23 +Secret access is on a "need to have" basis. If you think you need access to the

24 +secrets, please add your key to the `.sops.yaml` file on a PR and ping people

25 +that already have access for them to run the `updatekeys` command.

non-critical-infra/hosts/umbriel.nixos.org/README.md

+3 -3

@@ -2,9 +2,9 @@

3 ## Provisioning

5 -If you recreate `umbriel`, it will generate a new `DKIM` signature. That's

6 -ok to do, but you'll need to update the corresponding `mail._domainkey.*` `TXT`

7 -DNS record in `terraform/dns.tf` with the generated key in

5 +If you recreate `umbriel`, it will generate a new `DKIM` signature. That's ok to

6 +do, but you'll need to update the corresponding `mail._domainkey.*` `TXT` DNS

7 +record in `terraform/dns.tf` with the generated key in

8 `/var/dkim/mail-test.nixos.org.mail.txt`.

10 TODO: declaratively manage the `DKIM` key once

non-critical-infra/modules/mailserver/README.md

+2 -1

@@ -5,7 +5,8 @@ This module will [eventually][issue 485] provide mail services for `nixos.org`.

5 ## Mailing lists

7 To create a new mailing list, or change membership of a mailing list, see the

8 -instructions under `### Mailing lists go here ###` in [`default.nix`](./default.nix).

8 +instructions under `### Mailing lists go here ###` in

9 +[`default.nix`](./default.nix).

11 Some mailing lists allow login and sending email via `SMTP`. Search for

12 `loginAccount` to find examples of this.

terraform-iam/README.md

+8 -6

@@ -3,9 +3,10 @@

3 This module is for superadmins in the team.

5 This terraform root module manages:

6 -* IAM roles

7 -* fastly log module

8 -* infrastructure for archeologist team

6 +

7 +- IAM roles

8 +- fastly log module

9 +- infrastructure for archeologist team

11 ## Setup

@@ -17,7 +18,8 @@ Run `aws sso login` to acquire a temporary token.

19 ## Usage

20 -We use opentofu, which is a fork of https://www.terraform.io/ maintained by the Linux foundation.

21 +We use opentofu, which is a fork of https://www.terraform.io/ maintained by the

22 +Linux foundation.

24 Then run the following command to diff the changes and then apply if approved:

@@ -31,7 +33,7 @@ Write the Tofu code and test the changes using `./tf.sh validate`.

34 Before committing run `nix fmt`.

34 -Once the code is ready to be deployed, create a new PR with the attached

35 -output of `./tf.sh plan`.

36 +Once the code is ready to be deployed, create a new PR with the attached output

37 +of `./tf.sh plan`.

39 Once the PR is merged, run `./tf.sh apply` to apply the changes.

terraform/README.md

+14 -12

@@ -1,19 +1,20 @@

1 # For the bits that are not nixops-able

3 This terraform root module manages:

4 -* the resource in the AWS main account (S3 buckets)

5 -* Fastly

6 -* Netlify DNS

4 +

5 +- the resource in the AWS main account (S3 buckets)

6 +- Fastly

7 +- Netlify DNS

9 ## Setup

11 In order to use this, make sure to install direnv and Nix with flakes enabled.

12 -Then copy the `.envrc.local.template` to `.envrc.local`, and fill in the

13 -related keys.

13 +Then copy the `.envrc.local.template` to `.envrc.local`, and fill in the related

14 +keys.

15 -> FIXME: Unset the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars if

16 -> they are already set. Those have been replaced by AWS SSO.

16 +> FIXME: Unset the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars if they

17 +> are already set. Those have been replaced by AWS SSO.

19 Then run `direnv allow` to load the environment with the runtime dependencies.

@@ -21,7 +22,8 @@ Run `aws sso login` to acquire a temporary token.

23 ## Usage

24 -We use opentofu, which is a fork of https://www.terraform.io/ maintained by the Linux foundation.

25 +We use opentofu, which is a fork of https://www.terraform.io/ maintained by the

26 +Linux foundation.

28 Then run the following command to diff the changes and then apply if approved:

@@ -35,12 +37,12 @@ Write the Tofu code and test the changes using `./tf.sh validate`.

38 Before committing run `nix fmt`.

38 -Once the code is ready to be deployed, create a new PR with the attached

39 -output of `./tf.sh plan`.

40 +Once the code is ready to be deployed, create a new PR with the attached output

41 +of `./tf.sh plan`.

43 Once the PR is merged, run `./tf.sh apply` to apply the changes.

45 ## Upgrade from terraform to opentofu

45 -If you have used terraform, you may have to delete .terraform in this directory once

46 -to fixup provider registry addresses.

47 +If you have used terraform, you may have to delete .terraform in this directory

48 +once to fixup provider registry addresses.