| 1 | # 2024-06-13 |
| 2 | |
| 3 | Attendees: hexa, vcunat, Julien (partially), Eelco |
| 4 | |
| 5 | ## Round table |
| 6 | |
| 7 | - Julien |
| 8 | - currently otherwise occupied |
| 9 | - wants to finish the Lime survey migration away from AWS EC2 to non-critical |
| 10 | infra |
| 11 | - vcunat |
| 12 | - Full disk on Haumea |
| 13 | - Checking on tarball mirroring service |
| 14 | - wasn't working for the last two weeks, we failed to notice |
| 15 | - issue in nixpkgs caused breakdown |
| 16 | - tending to the script and will merge the fixed version back |
| 17 | - hexa |
| 18 | - Haumea's backup location |
| 19 | - Super write-intensive |
| 20 | - Return to rsync.net |
| 21 | - tried updating delft/* to 24.05 but hydra wouldn't compile |
| 22 | - Eelco |
| 23 | - Interested in the cost-increase on the release bucket |
| 24 | - March 6xxx USD |
| 25 | - April 9700 USD |
| 26 | - May 8200 USD |
| 27 | - Still increasing as of June |
| 28 | - Need to move forward with the S3 Bucket (Cache & Releases) |
| 29 | - Move data into Glacier, would be cheaper there, but not accessible from |
| 30 | cache.nixos.org anymore |
| 31 | - Moving things out of Glacier expensive, cheaper when we batch requests and |
| 32 | request them for the next day or so |
| 33 | - Plan to move to Tigris data, they would give us a discount, and egress is |
| 34 | currently free |
| 35 | - Need to get the relevant people in a room to make a final decision |
| 36 | - Eelco |
| 37 | - Edef |
| 38 | - Jonas |
| 39 | - Infra Build (hexa, vcunat) |
| 40 | |
| 41 | ## Action items |
| 42 | |
| 43 | - Check Prometheus Alerting Pipeline, no Alerts since May 21 |
| 44 | - File issue about hydra/nix build failures |
| 45 | - Schedule call about S3 bucket decision with Eelco, Jonas, Infra-Build |
| 46 | |
| 47 | ## Full disk on haumea |
| 48 | |
| 49 | - The ZFS pool (1 TB) on Haumea has been running full in the last few days, |
| 50 | leading to the PostgreSQL database to be unavailable |
| 51 | - Multiple options |
| 52 | - Reducing number and frequency of snapshots |
| 53 | - 3x5m, 4x15m, 24x1h, 4x1d, 3x1w |
| 54 | - [vcunat] 5 minutes probably excessive |
| 55 | - Replace haumea with a machine with bigger disks |
| 56 | - AX101 ~100 EUR/Mo |
| 57 | - Long-term maybe prune Hydras database |
| 58 | - or set up a new database and copy only the config over |
| 59 | |
| 60 | - Spend some more time debugging the situation, if it doesn't work out go for a |
| 61 | bigger machine |
| 62 | |
| 63 | ## Acquire rsync.net account for database backups of haumea |
| 64 | |
| 65 | - Previously rsync.net, but Account was paid by Graham. He eventually deleted |
| 66 | that account |
| 67 | - Currently only backup location is on hexa's NAS at home |
| 68 | - Backup size is currently 1.7TiB |
| 69 | - At 1.2 Cents per GB/Month that would cost ~24 USD/Month for 2TiB |
| 70 | - https://www.rsync.net/signup/order.html |
| 71 | |
| 72 | ## E-Mail Alias Management |
| 73 | |
| 74 | - Rok would like access, so that he can switch around the alias on the |
| 75 | streamyard account that the Marketing team uses |
| 76 | - Resource currently managed by Infra-Build |
| 77 | - Not enough opinions, discuss in internal infra room instead |
| 78 | |
| 79 | ### Changelog |
| 80 | |
| 81 | - Non-Critical-Infra updated to NixOS 24.05 |
| 82 | - migrated to systemd initrd |
| 83 | - Local Postfix setup for mail delivery from vault.nixos.org |
| 84 | - Owncast instance at live.nixos.org was set up |
| 85 | - Synapse Reverse-Proxying uses Unix Domain Sockets now |