meeting-notes: add 2024-06-13
Martin Weinelt committed
Jun 13, 2024 at 18:53 UTC
b84d56967c2161665f6bda284d04e0aacf35b2c0
1 file changed
+72
docs/meeting-notes/2024-06-13.md
new
+72
@@ -0,0 +1,72 @@
1
+# 2024-06-13
2
+
3
+Attendees: hexa, vcunat, Julien (partially), Eelco
4
+
5
+## Round table
6
+
7
+- Julien
8
+ - currently otherwise occupied
9
+ - wants to finish the Lime survey migration away from AWS EC2 to non-critical infra
10
+- vcunat
11
+ - Full disk on Haumea
12
+ - Checking on tarball mirroring service
13
+ - wasn't working for the last two weeks, we failed to notice
14
+ - issue in nixpkgs caused breakdown
15
+ - tending to the script and will merge the fixed version back
16
+- hexa
17
+ - Haumea's backup location
18
+ - Super write-intensive
19
+ - Return to rsync.net
20
+ - tried updating delft/* to 24.05 but hydra wouldn't compile
21
+- Eelco
22
+ - Interested in the cost-increase on the release bucket
23
+ - March 6xxx USD
24
+ - April 9700 USD
25
+ - May 8200 USD
26
+ - Still increasing as of June
27
+ - Need to move forward with the S3 Bucket (Cache & Releases)
28
+ - Move data into Glacier, would be cheaper there, but not accessible from cache.nixos.org anymore
29
+ - Moving things out of Glacier expensive, cheaper when we batch requests and request them for the next day or so
30
+ - Plan to move to Tigris data, they would give us a discount, and egress is currently free
31
+ - Need to get the relevant people in a room to make a final decision
32
+ - Eelco
33
+ - Edef
34
+ - Jonas
35
+ - Infra Build (hexa, vcunat)
36
+
37
+## Action items
38
+- Check Prometheus Alerting Pipeline, no Alerts since May 21
39
+- File issue about hydra/nix build failures
40
+- Schedule call about S3 bucket decision with Eelco, Jonas, Infra-Build
41
+
42
+## Full disk on haumea
43
+- The ZFS pool (1 TB) on Haumea has been running full in the last few days, leading to the PostgreSQL database to be unavailable
44
+- Multiple options
45
+ - Reducing number and frequency of snapshots
46
+ - 3x5m, 4x15m, 24x1h, 4x1d, 3x1w
47
+ - [vcunat] 5 minutes probably excessive
48
+ - Replace haumea with a machine with bigger disks
49
+ - AX101 ~100 EUR/Mo
50
+ - Long-term maybe prune Hydras database
51
+ - or set up a new database and copy only the config over
52
+
53
+- Spend some more time debugging the situation, if it doesn't work out go for a bigger machine
54
+
55
+## Acquire rsync.net account for database backups of haumea
56
+- Previously rsync.net, but Account was paid by Graham. He eventually deleted that account
57
+- Currently only backup location is on hexa's NAS at home
58
+- Backup size is currently 1.7TiB
59
+- At 1.2 Cents per GB/Month that would cost ~24 USD/Month for 2TiB
60
+ - https://www.rsync.net/signup/order.html
61
+
62
+## E-Mail Alias Management
63
+- Rok would like access, so that he can switch around the alias on the streamyard account that the Marketing team uses
64
+- Resource currently managed by Infra-Build
65
+- Not enough opinions, discuss in internal infra room instead
66
+
67
+### Changelog
68
+- Non-Critical-Infra updated to NixOS 24.05
69
+ - migrated to systemd initrd
70
+- Local Postfix setup for mail delivery from vault.nixos.org
71
+- Owncast instance at live.nixos.org was set up
72
+- Synapse Reverse-Proxying uses Unix Domain Sockets now