@cryptotaxi247 / infra
Code Commits Issues Pulls Sessions CI/CD
main
main
md 2.53 KB
View source Raw

2024-06-27

Attendees: edef, hexa, vcunat, zimbatm

Round table

  • hexa

    • Large PostgreSQL snapshot sizes caused by autovacuuming likely rewriting Indices (https://github.com/NixOS/infra/issues/446)
    • Actionables:
    • Setup rsync.net account, so we can have a proper backup, and help hexa's pipe
    • Try lighter compression with lz4 because we are seeing CPU load bottlenecking
    • https://github.com/NixOS/infra/pull/447
    • Tried the limesurvey migration. Slightly cursed because NixOS 22.05. Upgrade path not clear because of incompatible DB versions. Might need a fresh instance after talking to the marketing team.

  • vcunat:

    • Haumea zrepl snapshot frequency to accomodate the smol pipe of hexa's backup target
    • DB crashed due to full disk and would stop Hydra from working

  • edef:

    • Discussed with tomberek and jonas with getting the Glacier copy started. For only large objects to keep it simple.
    • The release bucket traffic has grown again?
    • edef: it doesn't seem that sizable based on the graphs I am watching
    • hexa: did you see the chart Eelco posted? they looked worrying
    • edef: to the fastly endpoint
    • hexa: AWS
    • edef: (looking the AWS Price explorer) looks like 1000 USD/month (30 USD/day), not exploded
    • 2000/2010 style infra team
    • We get this software thrown over and shall run it
    • How can Hydra be made future-proof?
    • Who maintains Hydra? Who makes sure the software works for the infra stack we can provide?
    • hexa: Only Ericson updates Hydra to new Nix versions, probably for CA derivations, not much else is happening
    • vcunat: Scale has increased much over the years since Hydra was written, and it hasn't kept up
    • edef: too few people to commit and cover stuff
    • biggest issues:
      • queue-runner cannot compute runnables faster than they are getting consumed
      • hydra kept busy with expensive xz compression of all results it gets

  • jonas:

    • requester pay on the release S3 bucket?
    • last rollout resulted in 404 (silent 403s)
    • we use the same code as for the cache
    • edef: I tried the fastly code for the cache bucket. Tried it on a separate deployment. It doesn't appear to experience the same issues. Doesn't require a privileged token. Not sure how to further debug that.
    • could talk about tigris data
    • edef: let's get stuff in there
    • edef: need to talk to AWS for free egress
    • jonas: just the release bucket for now, because we have issues with it