| 1 | # 2024-02-08 |
| 2 | |
| 3 | Attendees: delroth, edolstra, hexa, JulienMalka, raitobezarius, vcunat, zimbatm |
| 4 | |
| 5 | ## [delroth] FYI on availability next few weeks |
| 6 | |
| 7 | - Traveling until mid-April, low availability, will be on JST timezone (UTC+9) |
| 8 | - Missing for the next 2 infra meetings |
| 9 | |
| 10 | ## [delroth] Backups situation |
| 11 | |
| 12 | - How do we backup haumea, long term? |
| 13 | - borgbackup isn't really a good fit for a 500GB Postgres DB. |
| 14 | - Currently: zrepl to my personal infra and hexa's, but that's obviously not a |
| 15 | good long term solution. |
| 16 | - Used to have backups to graham's rsync.net account, but that's broken since |
| 17 | mid-Jan. |
| 18 | - [raito] Have you ever tried pg_dump's optimized dump format? |
| 19 | - [delroth] Is it fast enough to do a daily dump? |
| 20 | - [raito] unsure, but there are ways to do incremental backups: |
| 21 | - pg_basebackup + pg_dump compressed format |
| 22 | |
| 23 | ## [hexa] Migration of Synapse from EMS |
| 24 | |
| 25 | - Apparently waiting for EMS to sort out removal of PII? |
| 26 | - [raito] As long as there's discussion happening between Graham and EMS we |
| 27 | probably don't have to care about this, the legacy hosting plan is not getting |
| 28 | cancelled. |
| 29 | - [raito] If anything goes wrong we'd likely get notified. |
| 30 | |
| 31 | ## [eelco] Move fastly log aggregator to pluto |
| 32 | |
| 33 | - This is currently running on Eelco's local machine which is suboptimal. |
| 34 | - Weekly script that takes Fastly logs and loads them into AWS Athena + |
| 35 | generates some aggregates. |
| 36 | - https://github.com/NixOS/infra/tree/master/metrics/fastly |
| 37 | - We will put that on the new Eris: Pluto |
| 38 | - [eelco] I will need to create an AWS IAM to bestow the adequate permissions to |
| 39 | enable the script to run on Pluto. |
| 40 | - [eelco] I just need read/write access to Athena and some S3 bucket. |
| 41 | - [delroth] Who is using this data? |
| 42 | - [eelco] You can see on that page that the reporting is generated via this |
| 43 | data |
| 44 | - PII data regarding access logs of cache.nixos.org |
| 45 | - [everyone] What kind of policy do we want regarding PII and the non-critical |
| 46 | infrastructure? e.g. new wiki access logs are available to the non-critical |
| 47 | infrastructure |
| 48 | - Let's take note of this, think about it for the next weeks |
| 49 | |
| 50 | ## [delroth, hexa] Machine changes |
| 51 | |
| 52 | Our spend on outdated AWS EC2 instances and EBS volumes is too high and we are |
| 53 | cutting back on our use of EC2 and instead renew our infra at Hetzner. |
| 54 | |
| 55 | - Reduce AWS spending |
| 56 | - Started pruning old snapshots and EBS volumes (e.g. nixos-webserver, old |
| 57 | nixos versions) |
| 58 | - [eelco] I think it should be fine to delete them. There's a small risk |
| 59 | there could be some historical data, for instance, our subversion repo |
| 60 | used to be there as well and the nix-dev mailing list too. In theory, we |
| 61 | have copies of all of that. |
| 62 | - [delroth] I might start an instance and extract the data out there |
| 63 | otherwise I will just delete it. |
| 64 | - [eelco] There was a lot of scratch space for something… I don't remember |
| 65 | it. |
| 66 | - [delroth] I think it was bastion and is now paused. |
| 67 | - Bastion is now stopped/paused |
| 68 | - [hexa] Migrated to Eris and now to Pluto |
| 69 | - [hexa] Channel scripts are running way faster |
| 70 | - [raito] :tada: |
| 71 | - Pinged survey.nixos.org owners (@garbas), to get the limesurvey instance |
| 72 | migrated to something more reasonable |
| 73 | - [hexa] $ 150 USD/mo |
| 74 | - [hexa] Proposal: Migrate to Hetzner Cloud for a fraction of the costs |
| 75 | - [delroth] I asked Julien to look into it |
| 76 | - [delroth] In general, it's open to anyone who are looking to do |
| 77 | non-critical work |
| 78 | - Archeology machine from the cache team |
| 79 | - [delroth] Jonas, can you look into the cost? And can we make it start |
| 80 | on-demand? |
| 81 | - [jonas] asking edef whether they can accomodate these changes] |
| 82 | - Hetzner machine renewal |
| 83 | - Phasing out eris.nixos.org (EX41S-SSD, Intel i7-6700, 64GB RAM, 2x 256GB |
| 84 | SATA) |
| 85 | - [hexa] Old hardware |
| 86 | - Created and deployed pluto.nixos.org (EX44, Intel i5-13500, 2x512GB NVME) |
| 87 | - [hexa] Slightly cheaper but modern hardware |
| 88 | - [hexa] Everything migrated except for monitoring |
| 89 | - [delroth] Some disentanglement required to migrate monitoring |
| 90 | |
| 91 | There's a potential of around $700/month of savings in all those operations. |
| 92 | That is, we're offsetting our whole current Hetzner spend with those AWS |
| 93 | savings. |
| 94 | |
| 95 | - [delroth] Future savings (more involved): |
| 96 | - [delroth] Two layers of storage for cache.nixos.org: warm paths on Hetzner |
| 97 | - [delroth] It might be easier to do that stuff on NixOS releases S3 bucket |
| 98 | (much smaller bucket) and it's costing ~1000 USD per month in **bandwidth** |
| 99 | |
| 100 | ## [julien] Opening non-critical to more members |
| 101 | |
| 102 | - [Julien] Idea of non-critical infra was to lower the barrier to entry, because |
| 103 | people could be trusted with less risky infra |
| 104 | - [Julien] I would like to post a Discourse post to look for new people who |
| 105 | might be interested to join the team |
| 106 | - [Julien] It seems like we have some issues open for non-critical infra and |
| 107 | let people to tackle them and could constitute a first project |
| 108 | - [delroth] |
| 109 | https://github.com/NixOS/infra/issues?q=is%3Aopen+is%3Aissue+label%3Anon-critical-infra |
| 110 | - [Julien] I think it's a good time to do such a post and reach out |
| 111 | - [Julien] I wanted to know with everyone if it was okay to invite new people |
| 112 | - [delroth/zimbatm] Yes |
| 113 | - [delroth] I think the most important thing is to know who will take care of |
| 114 | onboarding and leading the work |
| 115 | - [Julien] I am ready to handle the onboarding load and the lead, I would |
| 116 | prefer to manage newcomers rather than do all the stuff by myself |
| 117 | |
| 118 | ## [delroth, hexa] Deployment changes |
| 119 | |
| 120 | We removed nixops and deployment now happens from a `flake.nix`. The plan is to |
| 121 | go for colmena eventually. |
| 122 | |
| 123 | - Deployment via |
| 124 | `nixos-rebuild --flake .#<host> --target-host root@<host>.nixos.org |
| 125 | --use-substitutes switch` |
| 126 | - NixOps generated configuration was imported and is being migrated, for example |
| 127 | we: |
| 128 | - started using agenix for secrets management and imported existing secrets |
| 129 | - and migrated Network configuration to systemd-networkd/resolved |
| 130 | |
| 131 | ## [delroth, hexa] Infra Changelog |
| 132 | |
| 133 | - All machines are now running on NixOS 23.11 |
| 134 | - Migrated haumea's database to PostgreSQL 16 |
| 135 | - Align timezone across machines |
| 136 | - Fix backup of haumea's database |
| 137 | - zrepl to delroth and hexa |
| 138 | - rsync.net stopped working due to zrepl API version mismatch |
| 139 | - Enabled trimming and scrubbing on all ZFS pools |
| 140 | |
| 141 | - Fix the fastly-exporter deployment |
| 142 | - Migrated to nixpkgs module, which |
| 143 | [required its own fixes](https://github.com/NixOS/nixpkgs/pull/287348) |
| 144 | - Generated a new API token, the old one was invalid |
| 145 | - 📊 |
| 146 | [Dashboard](https://monitoring.nixos.org/grafana/d/SHjM6e-ik/fastly?orgId=1) |
| 147 | - Fixed |
| 148 | [race condition and world-writable state |
| 149 | file](https://github.com/packethost/prometheus-packet-sd/issues/15) upstream |
| 150 | in packet-sd |
| 151 | - Added alerting for |
| 152 | - Failed systemd units |
| 153 | - [Domain expiry](https://github.com/NixOS/infra/pull/249) within the next 30 |
| 154 | days |
| 155 | - Lazy loading of eval errors on hydra (Patch by @ajs124) |
| 156 | - Reduces page sizes on the common jobsets/evals by 15-20MB to a few kBs |
| 157 | - More work needed, because error logs are still being fetched from the DB, |
| 158 | just not rendered |
| 159 | - Services migrated to pluto.nixos.org |
| 160 | - channel-scripts/hydra-mirror |
| 161 | - netboot |
| 162 | - rfc39 |
| 163 | - Removed and refactored legacy code, e.g. |
| 164 | - hydra-provisioner |
| 165 | - delft/network.nix |