@cryptotaxi247 / infra-1 / commits / 7038cbde

meeting-notes: add 2024-03-07

Martin Weinelt committed Mar 7, 2024 at 18:50 UTC 7038cbde93958f32949dbf2ee8fed64e06f6ad5f
1 file changed +102
docs/meeting-notes/2024-03-07.md new
+102
@@ -0,0 +1,102 @@
1 +# 2024-03-07
2 +
3 +Attendees: hexa, vcunat, zimbatm (Jonas), Linus, Julien, Raito/Ryan, Jade (most of the time)
4 +
5 +## [hexa] arm64 hetzner machine config
6 +
7 +- Dump it into a new directory in the infra repo, allow infra-build to deploy
8 + - vcunat: There's an issue containing the bits of the configuration
9 + - vcunat: I assumed we wanted to migrate it directly to a new deployment system
10 + - hexa: delroth wanted to script out iPXE but this has not panned out yet, we discovered we had DHCP available, which is promising
11 +
12 +## [zimbatm] Round table
13 +What is on everyone's mind? What are your plans?
14 +
15 +- Linus:
16 + - Happy to help out with stuff, pairing on with anything
17 + - zimbatm: Do you think we should do a better presentation?
18 + - linus: I think that'd be good
19 +- hexa:
20 + - Looking at iPXE, hold us back the most right now
21 + - will coordinate with delroth, if he has already anything
22 + - Open to discuss the Ceph scenario
23 + - A lot of discussions ongoing with the self-hosted binary cache, that's good
24 + - We are running into questions that cannot be answered by anyone
25 + - What should be the availability?
26 + - What should be the durability?
27 + - Discussion running in circles right now
28 + - Form a tightr discussion group
29 + - So that you can identify the main points
30 + - And address them
31 + - And not run into circles
32 +- vcunat:
33 + - Continuously busy with staging iterations
34 + - Unblocking difficult to access machines, e.g. aarch64 machine
35 + - There's actually more of my machines in the infra and that also requires update
36 + - Small benchmarking machine that makes sense:
37 + - t2a
38 + - The point is to have consistent benchmarking data
39 + - Linus: we definitely don't have cloud VMs for benchmarking, we probably want dedicated hardware
40 + - zimbatm: could you potentially create a ticket to make an inventory of your machines?
41 + - vcunat: there's two machines: t2a and t4b only really
42 +- Julien:
43 + - *Short-term*: I would like to onboard more folks on non-critical infrastructure
44 + - I would like to give them tasks to do end to end
45 + - Difficult to do with the current list of tasks atm
46 + - The wiki is also something I also want to get out ASAP
47 + - The technical issues are basically non-existent, just a little bit more work to do
48 + - Then announcements, onboard people to do editorial work, and that's it
49 + - We are near ready to launch
50 + - zimbatm: Bitwarden
51 + - Julien: we need to move the data from old to new and inform the change to the users
52 + - zimbatm: OK, we need to organize that migration
53 + - Julien: we can discuss this async
54 + - Interested also in cache self-hosting discussions
55 + - We have momentum and it'd be nice to have some sort of stance from infra people
56 + - Addressing the recent unrest regarding the public stance of infra on self hosting
57 + - zimbatm: we should/could do a proof of concept so we can get a feeling about how easy is it to operate
58 +- Ryan:
59 + - Recommend https://github.com/zhaofengli/colmena/pull/198
60 +
61 +Things to pick up for infra: https://github.com/NixOS/infra/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Anew-service
62 +
63 +## [hexa] darwin access
64 +
65 +- hexa: We have an inventory problem
66 + - What machines exist? What machines should we be able to access?
67 + - Important so we can delegate access and unblock work
68 + -
69 +
70 +- Braindump
71 + - Apple M1 at Hetzner (hydra)
72 + - Apple M1 in Grahams basement (???)
73 + - Apple M1 at Macstadium (ofborg)
74 + - Apple x86_64 at Macstadium (ofborg)
75 +
76 +## [hexa] ofBorg access
77 +
78 +- hexa: we have some folks who want to work on OfBorg but cannot do because they are not empowered on to do so
79 + - it is also go via buildkite management mechanism from Graham
80 +
81 +## [raito] aarch64.nixos.community management
82 +
83 +- https://github.com/NixOS/aarch64-build-box/
84 + - managed by community or infra?
85 + - zimbatm: it used to be in the nix-community infra, but because the nix-community does not have access to the Packet account
86 + - hexa: in the past, the worst we had is to debug the kernel issues, which is difficult w/o packet access
87 + - utilized by ofBorg, too, not a problem because we don't need to trust its build results
88 + - zimbatm: will talk with zowoq, who manages the nix-community day-to-day operation
89 +
90 +## Changelog
91 +- Cancelled the contract for `eris.nixos.org` (ends after 2024-02-28)
92 + - All services have been migrated to pluto.nixos.org
93 +- Set up backups for Prometheus, Grafana, VictoriaMetrics
94 +- The primary hostnames for Prometheus and Grafana have changed
95 + - https://prometheus.nixos.org
96 + - https://grafana.nixos.org
97 + - Redirects for the old hostname/path are in place
98 +- Hydra changes
99 + - Increase pipe size to improve queue-runner performance
100 + - Increased retention interval of Prometheus to two years so we have more history to evaluate these changes
101 +- Builders have received the fix for https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37
102 +- GitHub App for wiki.nixos.org so users can log in.