| 1 | # 2024-11-14 |
| 2 | |
| 3 | Attendees: jkarni, zimbatm, mic92, infinisil, kenji, drig/erethon, arian, sam , |
| 4 | hexa, jeremy, jeff |
| 5 | |
| 6 | ## Round Table |
| 7 | |
| 8 | ### Ofborg |
| 9 | |
| 10 | - Mic92: POC to evaluate nixpkgs on GitHub Actions. Results looked promising. |
| 11 | nixpkgs-review would run in 5 minutes. the ofborg-eval was heavily swapping |
| 12 | and taking 15min. |
| 13 | - Infinisil: people might have to enable GHA in their fork, which is disabled by |
| 14 | default |
| 15 | - Mic92: I didn't see this behaviour? |
| 16 | - Kenji: I think Github changed some defaults |
| 17 | - ref: |
| 18 | https://github.blog/changelog/2024-11-05-notice-of-breaking-changes-for-github-actions/#changes-to-workflow-validation-for-pull-requests-originating-from-forked-repositories |
| 19 | - Mic92: I think this is true for periodic |
| 20 | - Infinisil: trying now for a new user |
| 21 | - Mic92: if we want to pursue GHA, we would have to evaluate nixpkgs twice |
| 22 | because we need to get the store paths for master, and the changes of the PR, |
| 23 | and then we can compute all packages that have been changed, and append that |
| 24 | textfile as data. This can then be re-used by nixpkgs-review. |
| 25 | - Arian: why are we not using the PR workflow? |
| 26 | - Mic92: concurrency issues (limit of 20 runners per org). |
| 27 | - TODO: check if we hit the limit |
| 28 | - hexa: the idea to comment on the PR is to compensate for the visibility issue? |
| 29 | - Mic92: yes. it sucks a bit, but this could be mitigated by a small web |
| 30 | service. |
| 31 | - A thin wrapper that receives a webhook, checks back the PR status and |
| 32 | translates it as a comment. |
| 33 | - Infinisil: wouldn't it be possible to have a workflow that polls on behalf of |
| 34 | the user? |
| 35 | - It's a workflow that tries to find the workflow on the user's push, in their |
| 36 | fork. |
| 37 | - It would be triggered every time you synchronize the PR. |
| 38 | - Mic92: can you set this up in a way that the workflow gets triggered once |
| 39 | the workflow is finished? |
| 40 | - Infinisil: I think you need to poll for this. |
| 41 | - Mic92: Is it a 1:1 mapping, or 1:N? |
| 42 | - infinisil: Something like |
| 43 | ```yaml |
| 44 | # .github/workflows/query-pr.yml |
| 45 | on: pull_request_target |
| 46 | jobs: |
| 47 | check: |
| 48 | runs-on: ubuntu-latest |
| 49 | steps: |
| 50 | - run: | |
| 51 | gh api /repos/BASE_REPO/commits/GITHUB_SHA/check-runs |
| 52 | ``` |
| 53 | - Worry: Offloading OfBorg on to GH could give us trouble, because it might |
| 54 | not be insignificant compute. |
| 55 | - Jeremy: is this confined to PRs, or running on all branches? |
| 56 | - Mic92: it would be on push, but checking if the branch is part of the PR. |
| 57 | - Mic92: actually, there might be some synchronicity issue, because the PR |
| 58 | happens after the push. |
| 59 | - Mic92: Because of that we might need a webservice that can trigger actions |
| 60 | - Mic92: can we get an event when we open a PR? |
| 61 | |
| 62 | - Arian: Team plan gives us 60 concurrent actions by the way. (And team plan is |
| 63 | free for non-profit orgs) |
| 64 | |
| 65 | - Pushes don't have a base branch, need a base branch to compare the out paths |
| 66 | - Mic92: only if not open as a PR |
| 67 | - infinisil: Can pre-compute the out paths on push, cache out paths on Nixpkgs |
| 68 | master, then comparing can be done in a PR action fairly easily |
| 69 | |
| 70 | - Jonas: who is going to make this happen? |
| 71 | - Infinisil, Alex Balsoft, Jeremy after early December, Mic92 can write some |
| 72 | scripts and don't want to lead (want to work on the binary cache). |
| 73 | |
| 74 | - Infinisil: not convinced if that's a good idea. |
| 75 | |
| 76 | - Jonas: What would it take to get to feature parity |
| 77 | - What is the minimal set? |
| 78 | - Mic92: Minimal - Evaluate Prs |
| 79 | - Mic92: 2nd Phase - We can build packages. |
| 80 | - Mic92: Labels for mass-rebuilds |
| 81 | - Silvan: Requests reviews from maintainers (maybe not needed?) |
| 82 | - Silvan: Don't need to build manual with OfBorg anymore (Is already built in |
| 83 | ci) |
| 84 | - Silvan: Evaluating without aliases |
| 85 | - Jonas: Discourage IFD's |
| 86 | - Silvan: Maybe it really is good to split this up in two parts: |
| 87 | - Evaluating |
| 88 | - Building |
| 89 | |
| 90 | - Arian: Average job queue time is currently 9s: |
| 91 | https://github.com/NixOS/nixpkgs/actions/metrics/performance |
| 92 | - Arian: I would aim for: Lets just try with `pull_request:` and only do the |
| 93 | complicated `push:` abuse if that job queue time is gonna go up significantly |
| 94 | - Jonas: Looking forward to having eval failures to block merges! |
| 95 | |
| 96 | - Silvan: Who can review pr's and help out: |
| 97 | - kenji: +1 |
| 98 | - balsoft: +1 |
| 99 | - Mic92: +1 |
| 100 | - dgrig/erethon: +1 |
| 101 | - Silvan: Can add GH Team to ping for this issue |
| 102 | - Silvan: TODO - Mention this effort on Discourse |
| 103 | |
| 104 | - Mic92: How do we coordinate? |
| 105 | - Main Evalation |
| 106 | - Figure out parts we can parallelize: |
| 107 | - most parts are fairly orthogonal |
| 108 | |
| 109 | - Silvan: Somebody could lead the Building part: |
| 110 | - Mic92: Find someone who can help out, maybe on discourse? |
| 111 | - Silvan: GH doesn't have all the architectures |
| 112 | - Mic92: Start with the ones we have currently |
| 113 | - Silvan: If we don't need ealuation anymore - this could save a lot of |
| 114 | resources, could optimize |
| 115 | - hexa: Yes, but it might not apply on top of staging for example |
| 116 | - Silvan: Yes, staging can probably be ignored |
| 117 | - hexa: Yes, it tries to build against the target branch, led to some |
| 118 | problems, for example always trying to build llvm on darwin -> continuuous |
| 119 | timeouts |
| 120 | - Mic92: A ton of stuff we could potentially optimize |
| 121 | - Silvan: Empower users to build on more architectures |
| 122 | - Mic92: Convenient to have logs in public |
| 123 | - Mic92: I would like to see a /build command, so that builds can be manually |
| 124 | triggered |
| 125 | |
| 126 | - Silvan: Optimization of the Eval part: |
| 127 | - look at path that actually changed |
| 128 | - mic92: Aware of nix script that gives names of paths that are actually |
| 129 | changed? |
| 130 | - Silvan: Yes |
| 131 | - Mic92: If heavy swapping, then it might speed up, else we might see a slow |
| 132 | down |
| 133 | |
| 134 | - Silvan: Where should we report? |
| 135 | - discourse? |
| 136 | - Mic92: Discussion would be nicer on GH, because we can link issues/pr's. |
| 137 | |
| 138 | ## Topics |
| 139 | |
| 140 | - Transfer of the Macs located at Detsys to Flying Circus |
| 141 | - Scheduled for 2024-11-25 |
| 142 | - Currently enrolled into Detsys MDM Account. Can we set something up to |
| 143 | migrate that to an infra team account? |
| 144 | - Report back the result to the Mac Mini Logistics room on Matrix |
| 145 | - MDM built into macs, but need to be enrolled into an mdm vendor |
| 146 | - Arian: Don't have to do it, but very convenient |
| 147 | - Mic92: if there are no major problems, should look into MDM as well. |
| 148 | - Oakhost Macs are available |
| 149 | - Need the usual setup |
| 150 | - 3 new machines |
| 151 | - arian can set this up |
| 152 | - initial password |
| 153 | - Mic92: Mac enrollment not very automated yet, last time hexa wrote some |
| 154 | stuff down |
| 155 | - Arians keys needs to be added to the repository |
| 156 | - Arian: |
| 157 | - need access to oakhost |
| 158 | - ssh key to infra |
| 159 | - Mac Issues: |
| 160 | - hexa: Forking issues on seqoia |
| 161 | - hexa: Running quite well atm |
| 162 | - hexa: Patched out chrooting of nix, applied patches on top of darwin |
| 163 | builders |
| 164 | - hexa: Not sure exactly why that works |
| 165 | - hexa: Upstreaming rosetta2-gc to nix-darwin currently |
| 166 | - hexa: darwin 15.1 had issues -> hetzner doesn't roll back (need rescue mode) |
| 167 | - hexa: Rollbacks likely possible with mdm |
| 168 | - arian: We can likely add hetzner darwin machines to be managed by mdm, but |
| 169 | not too sure |
| 170 | - arian: Looks into if we can add without physical access, look into what |
| 171 | detsys did |
| 172 | - Equinix Metal Exit Plan |
| 173 | - https://md.darmstadt.ccc.de/eqm-exit-plan |
| 174 | - hexa: This is what we had, this is what we need, this is what it is going to |
| 175 | cost. |
| 176 | - mic92: Should we check out what we need for the arm64 builders? |
| 177 | - hexa: Basically choice between: 64GB, or 256GB of memory |
| 178 | - hexa: Likely want the bigger memory |
| 179 | - Mic92: Can try the same for arm64, for x86 we can look in to funding. |
| 180 | - Mic92: How long do we need for set up? - a day? Shouldn't take long to set |
| 181 | up. |
| 182 | - Mic92: I will set this in motion, unless someone else want's to reach out. |
| 183 | - Mic92: Ok, I will do it. |
| 184 | - hexa: Ideally we don't have 20 small machine, but 5 big machines. Which |
| 185 | would be good for maintenance reasons. Because we likely won't get a netboot |
| 186 | setup anymore. |
| 187 | - Security Tracker |
| 188 | - dgrig: Jonas gave me access to a Hetzner Cloud project a couple of weeks |
| 189 | ago. A VM is up and running, I'm figuring out how to implement this in the |
| 190 | same way as nixos-infra. |
| 191 | - dgrig: Do we care about having this in Terraform? I used TF to spin this up, |
| 192 | but the state is in my computer currently, do we care to push it to S3? |
| 193 | - drgrig: Can I do `nixos-installer --flake nixos/nixos-infra`?How can I |
| 194 | install the nixos-infra |
| 195 | - Mic02: Inputs :"${inputs.nixos-infra}/keys" can convert to string. |
| 196 | - Example: |
| 197 | ```nix |
| 198 | users.users.root.openssh.authorizedKeys.keys = [] ++ (builtins.filter (l: l != [ ]) (builtins.split "\n" (builtins.readFile inputs.phaer-keys))); |
| 199 | ``` |
| 200 | - dgrig: Do we care about the Terraform state yet? |
| 201 | - hexa: Do we need the state yet? |
| 202 | - drgrid: It is the tf state |
| 203 | - consensus: We don't care |
| 204 | - Jeremy: Nit - export the keys as a module |