| 1 | # 2024-06-27 |
| 2 | |
| 3 | Attendees: edef, hexa, vcunat, zimbatm |
| 4 | |
| 5 | ## Round table |
| 6 | |
| 7 | - hexa |
| 8 | - Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting |
| 9 | Indices (https://github.com/NixOS/infra/issues/446) |
| 10 | |
| 11 | - Actionables: |
| 12 | 1. Setup rsync.net account, so we can have a proper backup, and help hexa's |
| 13 | pipe |
| 14 | 2. Try lighter compression with lz4 because we are seeing CPU load |
| 15 | bottlenecking |
| 16 | 3. https://github.com/NixOS/infra/pull/447 |
| 17 | - Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade |
| 18 | path not clear because of incompatible DB versions. Might need a fresh |
| 19 | instance after talking to the marketing team. |
| 20 | |
| 21 | - vcunat: |
| 22 | - Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup |
| 23 | target |
| 24 | - DB crashed due to full disk and would stop Hydra from working |
| 25 | |
| 26 | - edef: |
| 27 | - Discussed with tomberek and jonas with getting the Glacier copy started. For |
| 28 | only large objects to keep it simple. |
| 29 | - The release bucket traffic has grown again? |
| 30 | - edef: it doesn't seem that sizable based on the graphs I am watching |
| 31 | - hexa: did you see the chart Eelco posted? they looked worrying |
| 32 | - edef: to the fastly endpoint |
| 33 | - hexa: AWS |
| 34 | - edef: (looking the AWS Price explorer) looks like 1000 USD/month (30 |
| 35 | USD/day), not exploded |
| 36 | - 2000/2010 style infra team |
| 37 | - We get this software thrown over and shall run it |
| 38 | - How can Hydra be made future-proof? |
| 39 | - Who maintains Hydra? Who makes sure the software works for the infra stack |
| 40 | we can provide? |
| 41 | - hexa: Only Ericson updates Hydra to new Nix versions, probably for CA |
| 42 | derivations, not much else is happening |
| 43 | - vcunat: Scale has increased much over the years since Hydra was written, |
| 44 | and it hasn't kept up |
| 45 | - edef: too few people to commit and cover stuff |
| 46 | - biggest issues: |
| 47 | - queue-runner cannot compute runnables faster than they are getting |
| 48 | consumed |
| 49 | - hydra kept busy with expensive xz compression of all results it gets |
| 50 | |
| 51 | - jonas: |
| 52 | - requester pay on the release S3 bucket? |
| 53 | - last rollout resulted in 404 (silent 403s) |
| 54 | - we use the same code as for the cache |
| 55 | - edef: I tried the fastly code for the cache bucket. Tried it on a separate |
| 56 | deployment. It doesn't appear to experience the same issues. Doesn't |
| 57 | require a privileged token. Not sure how to further debug that. |
| 58 | - could talk about tigris data |
| 59 | - edef: let's get stuff in there |
| 60 | - edef: need to talk to AWS for free egress |
| 61 | - jonas: just the release bucket for now, because we have issues with it |