meeting-notes: add 2024-06-07
Martin Weinelt committed
Jun 27, 2024 at 20:29 UTC
8ff24cb2648f7f08749d5891aa9b0bacb638fe84
1 file changed
+47
docs/meeting-notes/2024-06-27.md
new
+47
@@ -0,0 +1,47 @@
1
+# 2024-06-27
2
+
3
+Attendees: edef, hexa, vcunat, zimbatm
4
+
5
+## Round table
6
+
7
+- hexa
8
+ - Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting Indices (https://github.com/NixOS/infra/issues/446)
9
+
10
+ - Actionables:
11
+ 1. Setup rsync.net account, so we can have a proper backup, and help hexa's pipe
12
+ 2. Try lighter compression with lz4 because we are seeing CPU load bottlenecking
13
+ 3. https://github.com/NixOS/infra/pull/447
14
+ - Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade path not clear because of incompatible DB versions. Might need a fresh instance after talking to the marketing team.
15
+
16
+- vcunat:
17
+ - Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup target
18
+ - DB crashed due to full disk and would stop Hydra from working
19
+
20
+- edef:
21
+ - Discussed with tomberek and jonas with getting the Glacier copy started. For only large objects to keep it simple.
22
+ - The release bucket traffic has grown again?
23
+ - edef: it doesn't seem that sizable based on the graphs I am watching
24
+ - hexa: did you see the chart Eelco posted? they looked worrying
25
+ - edef: to the fastly endpoint
26
+ - hexa: AWS
27
+ - edef: (looking the AWS Price explorer) looks like 1000 USD/month (30 USD/day), not exploded
28
+ - 2000/2010 style infra team
29
+ - We get this software thrown over and shall run it
30
+ - How can Hydra be made future-proof?
31
+ - Who maintains Hydra? Who makes sure the software works for the infra stack we can provide?
32
+ - hexa: Only Ericson updates Hydra to new Nix versions, probably for CA derivations, not much else is happening
33
+ - vcunat: Scale has increased much over the years since Hydra was written, and it hasn't kept up
34
+ - edef: too few people to commit and cover stuff
35
+ - biggest issues:
36
+ - queue-runner cannot compute runnables faster than they are getting consumed
37
+ - hydra kept busy with expensive xz compression of all results it gets
38
+
39
+- jonas:
40
+ - requester pay on the release S3 bucket?
41
+ - last rollout resulted in 404 (silent 403s)
42
+ - we use the same code as for the cache
43
+ - edef: I tried the fastly code for the cache bucket. Tried it on a separate deployment. It doesn't appear to experience the same issues. Doesn't require a privileged token. Not sure how to further debug that.
44
+ - could talk about tigris data
45
+ - edef: let's get stuff in there
46
+ - edef: need to talk to AWS for free egress
47
+ - jonas: just the release bucket for now, because we have issues with it