docs: add infra meeting notes
Pierre Bourdon committed
Feb 12, 2024 at 22:54 UTC
43042cfa9af66d6b42d3596435b85dcadb787d2d
3 files changed
+203
docs/meeting-notes/2024-01-11.md
new
+61
@@ -0,0 +1,61 @@
1
+# 2024-01-11
2
+
3
+First meeting of the (revamped) infra team.
4
+
5
+Participants: delroth, hexa, raitobezarius, vcunat, zimbatm
6
+
7
+## [zimbatm] Presentation
8
+
9
+- At NixCon, we added new people to the team, but we were not able to give space to those new
10
+ people, with this in mind, I would like to dedicate one hour per week or two weeks where I can
11
+ unblock the infrastructure matters.
12
+- I don’t know what people are interested in, I believe this is a volunteer ecosystem and you should
13
+ work on what you would like to work on.
14
+- We have big challenges in front of us, e.g. the cache situation, with a new team, maybe we can
15
+ tackle those bigger challenges.
16
+
17
+## Round of intros
18
+
19
+Skipped in these edited notes.
20
+
21
+## [raito] Recommending hexa for infra-core
22
+
23
+- Consensus: yes please.
24
+- [zimbatm] Done.
25
+- delroth/vcunat to assist with onboarding, provision access, etc.
26
+
27
+## [delroth] Matrix Homeserver situation
28
+
29
+- EMS is dropping legacy plans after 2024-01-17
30
+ - https://github.com/NixOS/infra/issues/325
31
+- We are getting dropped.
32
+- We need to react but Graham, owner of the EMS account, is not reacting.
33
+- The problem is not the cost but access to the account.
34
+- delroth/hexa are in favor of self-hosting.
35
+ - But we need the database dump from EMS.
36
+ - hexa to prepare the config for this, delroth can act as backup/fallback.
37
+- Fallback: we can always pay the $1200 (excl. VAT) for renewing the 1 year plan.
38
+
39
+## [hexa] Moving NGI out of nixos-org-configurations
40
+
41
+- Goals: unblock ngi0 maintainers, less consumption of our review bandwidth.
42
+- Should we move them to a new repo?
43
+ - Either in the nixos GitHub org or the ngi-nix org.
44
+- Action item: let's ask them!
45
+ - https://github.com/NixOS/infra/issues/326
46
+
47
+## Builders
48
+
49
+- Context: various cost reduction efforts need to happen on the Hydra/ofborg builders infra.
50
+- There might be the possibility to get Hetzner to sponsor one more machine.
51
+- [delroth] Pretty sure we are not using our build resources efficiently as it is (queue-runner
52
+ bottleneck)
53
+- [vcunat] xz compression is the main problem
54
+- [zimbatm] We should properly analyze where the bottlenecks are.
55
+
56
+## Backups
57
+
58
+- We are not doing proper backups of the NixOS infra.
59
+- There is an rsync.net account where the Hydra database gets backed up to, at least.
60
+- Julien's vaultwarden PR is currently blocked by this, we're getting backup storage space from
61
+ Hetzner (storage boxes).
docs/meeting-notes/2024-01-25.md
new
+41
@@ -0,0 +1,41 @@
1
+# 2024-01-25
2
+
3
+## [hexa, delroth] EMS Migration
4
+
5
+- Configuration hasn’t been written yet, hexa might get it done this week.
6
+- When will we get the data?
7
+ - Graham still holding it until it can get cleaned up (removing private user data). Board set a
8
+ deadline during the last meeting.
9
+ - We could talk to EMS directly, to get the account handed over
10
+ - We want ~10 days to do the migration (so: we want the data before Feb 7th)
11
+
12
+## NixOS 23.11 upgrades
13
+
14
+- Infra currently runs on NixOS 23.05
15
+- No blockers, need to be updated individually
16
+
17
+## Deployment setup
18
+
19
+- Blocked on secret management, will likely be sops
20
+- Machines use network configuration provided by NixOps
21
+
22
+## Bitwarden
23
+
24
+- Reason: Self-hosting, currently Jonas pays for the hosted plan.
25
+- PR pending needs to be moved forward: https://github.com/NixOS/infra/pull/287
26
+- delroth/hexa can hand out backup storage credentials.
27
+
28
+## Binary cache
29
+
30
+- Cost of S3 exceeds Foundation income…
31
+ - Garbage collection will be started
32
+ - Timeline: Start some time in 2024/02
33
+ - Advanced communication will be sent out
34
+ - Build list of store paths we want to keep and configure gc root for them
35
+ - Plan is to keep all FODs
36
+ - Make store paths that are about to get deleted unavailable prior to deletion
37
+- Potentially move parts of the cache to Hetzner
38
+ - delroth has capacity to look into this in 2024/02
39
+ - Needs a service to decide, where (S3 or Hetzner) the request would need to go
40
+ - Logic could be installed at fastly, to try hetzner first, fallback to s3
41
+ - Service is in the critical path, currently fastly/s3 solve availability for us
docs/meeting-notes/2024-02-08.md
new
+101
@@ -0,0 +1,101 @@
1
+# 2024-02-08
2
+
3
+Attendees: delroth, hexa, JulienMalka, lheckemann, raitobezarius, vcunat, zimbatm
4
+
5
+## [hexa, delroth] EMS Migration
6
+
7
+Context: https://github.com/NixOS/infra/issues/325
8
+
9
+- PR for Synapse and its dependencies is up.
10
+ - https://github.com/NixOS/infra/pull/336
11
+- [Julien] What's the status of the backup module?
12
+ - Split off into its own PR and merged already: https://github.com/NixOS/infra/pull/345
13
+- raito and Ron met with Matrix / EMS folks at FOSDEM 2024
14
+ - They have scripts for GDPR compliance (user data purge), but we need to ask them by email.
15
+ - Then we can get a clean DB dump, presumably without user data.
16
+ - Not sure whether we sent an email or not. But Graham might be in contact directly, and EMS folks
17
+ made him an offer to do the data deletion.
18
+ - Worst case Graham/DetSys will pay for the extension of the EMS plan.
19
+ - Probably no hurry anymore from the infra side. Foundation board is monitoring this to make sure
20
+ we have a solution at some point.
21
+
22
+## [delroth] Should we publish these notes more widely?
23
+
24
+- There is a trend towards publishing notes on Discourse, etc. for visibility.
25
+- [delroth] My thoughts: we should archive (edited) notes in Git somewhere in our docs/ folder,
26
+ update a Discourse thread every 2 weeks.
27
+ - I of course volunteer to take care of this :)
28
+- Consensus: let’s do it.
29
+
30
+## [delroth] Packet/EQM access to infra-core
31
+
32
+- Our builders are very, very outdated. But risky to try and update stuff with 0 debugging
33
+ capabilities.
34
+- Any reason why infra-core shouldn’t have full Packet/EQM access like we have Hetzner access?
35
+ - Not entirely clear who currently has access?
36
+ - [zimbatm] Got access from eelco last weekend, will delegate.
37
+- [raito] Does nix-netboot-serve run on our infra?
38
+ - [hexa] Yes, on eris. The images are also built from our infra, it’s a Hydra jobset. But the
39
+ jobset has not successfully completed for a year.
40
+ - [hexa] We can update stuff, but we have no way to debug issues if we do so.
41
+- zimbatm took care of it live, woo!
42
+
43
+## [raito] Stay in the loop of infrastructure matters
44
+
45
+- How should work be split between zimbatm/raito?
46
+- Would like access to private infra stuff to act as secondary.
47
+- In general: who should have ownership to accounts?
48
+ - A bunch of GH org owners for example are inactive.
49
+ - Not really aligned with any subgroup e.g. foundation board.
50
+ - [zimbatm] I think the foundation should have access, but unfortunately the foundation also
51
+ doesn’t have the best personal security to hold those credentials.
52
+ - [zimbatm] Maybe it should be the infra team instead? i.e. delroth/hexa/vcunat/…
53
+ - [raito] That would work too, as long as it’s active folks who can take care of day to day stuff.
54
+ I don’t care that it’s specifically me, just that we don’t get blocked due to not finding an
55
+ owner.
56
+ - [zimbatm] I don’t feel like I can make that decision alone right now. Let’s find some kind of
57
+ organization which makes sense.
58
+- Raito got invited into the private infra matrix channel (at least, for now)
59
+
60
+## [Julien] NixOS wiki collaboration w/ infra team
61
+
62
+- We have a bunch of candidate sysadmins in mind. Do we want to merge this into non-critical-infra?
63
+- [Julien] I’m a bit biased since I’m sitting on both sides of this discussion, but I think this
64
+ would be a good onramp to bring more people into non-critical-infra.
65
+- [zimbatm] We can subdivide permissions on the Hetzner Cloud side of things, but I’m not sure
66
+ whether we should share stuff further.
67
+- [hexa] They have their setup mostly figured out already, including backups. We can let them run
68
+ with it for now, and we can always pick it up later.
69
+- [linus] What about inviting them to non-critical-infra and just giving them access to all the
70
+ non-critical-infra? Even if they just want to maintain the wiki.
71
+ - [hexa] It’s about responsible for all of it. I don’t think we should grant unneeded access.
72
+ - [Julien] +1.
73
+ - [delroth] I feel like if it’s official, we should treat it as such and onboard it as part of
74
+ non-critical infra. Doesn’t require giving them access to everything.
75
+ - [linus] If it is official, then it should be maintained by the official infra team
76
+ - [hexa] I think we’re mostly in agreement then.
77
+- [delroth] non-critical-infra should be restricted to the relevant directories and go through PRs
78
+ for touching other stuff
79
+ - [Julien] They probably want to iterate fast in the beginning
80
+ - [delroth] They should get a dedicated machine on Hetzner Cloud, that they can play with
81
+ - [Julien] Too much shared code will increase reliance on core infra members.
82
+- [delroth] Action items
83
+ - Let’s give them SSH access to a Hetzner Cloud VM
84
+ - Or a separate project so they get direct access to machines. Might already be done.
85
+ - Let’s make sure we agree on the idea of moving this to non-critical-infra in the short/mid-term
86
+ future
87
+ - Provision DNS etc.
88
+
89
+## External requests
90
+
91
+- Hydra DB access (raitobezarius)
92
+ - Hashing out details in https://github.com/NixOS/infra/issues/348
93
+- CA derivations for Hydra (Ericson2314)
94
+ - Nix 2.20 broke interop with the old Nix 2.13 we run on builders. Rolled back to 2.19.
95
+ - https://github.com/NixOS/nix/issues/9961
96
+ - DB schema change applied.
97
+
98
+## Ongoing projects
99
+
100
+- [delroth] Hoping to complete the nixops deprecation this week. Then: core/non-critical-infra
101
+ alignment.