@cryptotaxi247 / infra
Code Commits Issues Pulls Sessions CI/CD
main
main
md 2.9 KB
View source Raw

2024-06-13

Attendees: hexa, vcunat, Julien (partially), Eelco

Round table

  • Julien
    • currently otherwise occupied
    • wants to finish the Lime survey migration away from AWS EC2 to non-critical infra
  • vcunat
    • Full disk on Haumea
    • Checking on tarball mirroring service
    • wasn't working for the last two weeks, we failed to notice
    • issue in nixpkgs caused breakdown
    • tending to the script and will merge the fixed version back
  • hexa
    • Haumea's backup location
    • Super write-intensive
    • Return to rsync.net
    • tried updating delft/* to 24.05 but hydra wouldn't compile
  • Eelco
    • Interested in the cost-increase on the release bucket
    • March 6xxx USD
    • April 9700 USD
    • May 8200 USD
    • Still increasing as of June
    • Need to move forward with the S3 Bucket (Cache & Releases)
    • Move data into Glacier, would be cheaper there, but not accessible from cache.nixos.org anymore
    • Moving things out of Glacier expensive, cheaper when we batch requests and request them for the next day or so
    • Plan to move to Tigris data, they would give us a discount, and egress is currently free
    • Need to get the relevant people in a room to make a final decision
      • Eelco
      • Edef
      • Jonas
      • Infra Build (hexa, vcunat)

Action items

  • Check Prometheus Alerting Pipeline, no Alerts since May 21
  • File issue about hydra/nix build failures
  • Schedule call about S3 bucket decision with Eelco, Jonas, Infra-Build

Full disk on haumea

  • The ZFS pool (1 TB) on Haumea has been running full in the last few days, leading to the PostgreSQL database to be unavailable

  • Multiple options

    • Reducing number and frequency of snapshots
    • 3x5m, 4x15m, 24x1h, 4x1d, 3x1w
      • [vcunat] 5 minutes probably excessive
    • Replace haumea with a machine with bigger disks
    • AX101 ~100 EUR/Mo
    • Long-term maybe prune Hydras database
    • or set up a new database and copy only the config over

  • Spend some more time debugging the situation, if it doesn't work out go for a bigger machine

Acquire rsync.net account for database backups of haumea

  • Previously rsync.net, but Account was paid by Graham. He eventually deleted that account
  • Currently only backup location is on hexa's NAS at home
  • Backup size is currently 1.7TiB
  • At 1.2 Cents per GB/Month that would cost ~24 USD/Month for 2TiB

E-Mail Alias Management

  • Rok would like access, so that he can switch around the alias on the streamyard account that the Marketing team uses
  • Resource currently managed by Infra-Build
  • Not enough opinions, discuss in internal infra room instead

Changelog

  • Non-Critical-Infra updated to NixOS 24.05
    • migrated to systemd initrd
  • Local Postfix setup for mail delivery from vault.nixos.org
  • Owncast instance at live.nixos.org was set up
  • Synapse Reverse-Proxying uses Unix Domain Sockets now