@cryptotaxi247 / infra-1
Code Commits Issues Pulls Sessions CI/CD
main
main
md 4.91 KB
View source Raw

2024-03-07

Attendees: hexa, vcunat, zimbatm (Jonas), Linus, Julien, Raito/Ryan, Jade (most of the time)

[hexa] arm64 hetzner machine config

  • Dump it into a new directory in the infra repo, allow infra-build to deploy
    • vcunat: There's an issue containing the bits of the configuration
    • vcunat: I assumed we wanted to migrate it directly to a new deployment system
    • hexa: delroth wanted to script out iPXE but this has not panned out yet, we discovered we had DHCP available, which is promising

[zimbatm] Round table

What is on everyone's mind? What are your plans?

  • Linus:
    • Happy to help out with stuff, pairing on with anything
    • zimbatm: Do you think we should do a better presentation?
    • linus: I think that'd be good
  • hexa:
    • Looking at iPXE, hold us back the most right now
    • will coordinate with delroth, if he has already anything
    • Open to discuss the Ceph scenario
    • A lot of discussions ongoing with the self-hosted binary cache, that's good
    • We are running into questions that cannot be answered by anyone
      • What should be the availability?
      • What should be the durability?
      • Discussion running in circles right now
    • Form a tightr discussion group
      • So that you can identify the main points
      • And address them
      • And not run into circles
  • vcunat:
    • Continuously busy with staging iterations
    • Unblocking difficult to access machines, e.g. aarch64 machine
    • There's actually more of my machines in the infra and that also requires update
    • Small benchmarking machine that makes sense:
    • t2a
    • The point is to have consistent benchmarking data
    • Linus: we definitely don't have cloud VMs for benchmarking, we probably want dedicated hardware
    • zimbatm: could you potentially create a ticket to make an inventory of your machines?
    • vcunat: there's two machines: t2a and t4b only really
  • Julien:
    • Short-term: I would like to onboard more folks on non-critical infrastructure
    • I would like to give them tasks to do end to end
    • Difficult to do with the current list of tasks atm
    • The wiki is also something I also want to get out ASAP
    • The technical issues are basically non-existent, just a little bit more work to do
    • Then announcements, onboard people to do editorial work, and that's it
    • We are near ready to launch
    • zimbatm: Bitwarden
    • Julien: we need to move the data from old to new and inform the change to the users
    • zimbatm: OK, we need to organize that migration
    • Julien: we can discuss this async
    • Interested also in cache self-hosting discussions
    • We have momentum and it'd be nice to have some sort of stance from infra people
    • Addressing the recent unrest regarding the public stance of infra on self hosting
    • zimbatm: we should/could do a proof of concept so we can get a feeling about how easy is it to operate
  • Ryan:

Things to pick up for infra: https://github.com/NixOS/infra/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Anew-service

[hexa] darwin access

  • hexa: We have an inventory problem

    • What machines exist? What machines should we be able to access?

    - Important so we can delegate access and unblock work

  • Braindump

    • Apple M1 at Hetzner (hydra)
    • Apple M1 in Grahams basement (???)
    • Apple M1 at Macstadium (ofborg)
    • Apple x86_64 at Macstadium (ofborg)

[hexa] ofBorg access

  • hexa: we have some folks who want to work on OfBorg but cannot do because they are not empowered on to do so
    • it is also go via buildkite management mechanism from Graham

[raito] aarch64.nixos.community management

  • https://github.com/NixOS/aarch64-build-box/
    • managed by community or infra?
    • zimbatm: it used to be in the nix-community infra, but because the nix-community does not have access to the Packet account
    • hexa: in the past, the worst we had is to debug the kernel issues, which is difficult w/o packet access
    • utilized by ofBorg, too, not a problem because we don't need to trust its build results
    • zimbatm: will talk with zowoq, who manages the nix-community day-to-day operation

Changelog

  • Cancelled the contract for eris.nixos.org (ends after 2024-02-28)
    • All services have been migrated to pluto.nixos.org
  • Set up backups for Prometheus, Grafana, VictoriaMetrics
  • The primary hostnames for Prometheus and Grafana have changed
  • Hydra changes
    • Increase pipe size to improve queue-runner performance
    • Increased retention interval of Prometheus to two years so we have more history to evaluate these changes
  • Builders have received the fix for https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37
  • GitHub App for wiki.nixos.org so users can log in.