@cryptotaxi247 / infra-1 / commits / e6460c55

Bastion: rescue!

When moving away from NixOps, it looks like we missed importing the amazon-image.nix, and the bootloader config which was added to the configuration.nix was a bit wrong, pointing grub to `nodev` when it should have been `/dev/xvda1`. Rescuing this image involved: 1. Snapshotted the disk 1. Shut down the (broken) bastion 1. Booted another NixOS machine 1. Detached /dev/sda1 (note: the sda*1* here is not a typo!) from within the EC2 Console 1. Attached the root disk to the new machine as /dev/sdf, which appeared to as /dev/nvme1n1 Then, `lsblk` and `mount` showed the NEW DISK being mounted as the root and /nix/store, but these tools were *lying*. Both disks had their root partition labeled "nixos". To fix this, we: 1. Detached the new disk from the rescue machine 1. Edited the rescue machine's /etc/nixos/configuration.nix to include: fileSystems."/".device = lib.mkForce "/dev/disk/by-label/rescue"; 1. Ran `nixos-rebuild boot` 1. Updated the rescue machine's root filesystem label to `e2label /dev/nvme0n1p2 rescue` 1. Rebooted and confirmed the root filesystem was mounted from by-label/rescue. Now, we re-stared the rescue process. 1. Attach the old machine's disk to the rescue machine 1. Mounted /dev/nmvme1n1p1 to /mnt 1. Made the changes in this commit, *EXCEPT* we also made the following change: boot.loader.grub.device = lib.mkForce "/dev/nvme1n1"; Note the `nvme1n1` was correct *for the rescue environment* but is not correct in any other environment. After rescuing the image and getting it booted, we deleted that line. We considered using /dev/disk/by-id instead, which probably would have been a good choice. We didn't use it just because we didn't. 1. nix-shell -p git nixUnstable 1. Inside this shell we ran `nixos-install --flake .#bastion` from `/mnt/home/deploy/src/nixos-org-configurations/bastion`. 1. Then we unmounted /mnt, detached the volume, and reattached the volume to the old machine and it booted correctly.

Graham Christensen committed Dec 17, 2021 at 21:35 UTC e6460c55f6776d0a12c0206a02b4f37d939e7d15
1 file changed +4 -9
bastion/configuration.nix
+4 -9
@@ -1,6 +1,6 @@
1 flakes @ { self, nixpkgs, nix, nixops, nixos-channel-scripts }:
2
3 -{ config, lib, pkgs, ... }:
3 +{ modulesPath, config, lib, pkgs, ... }:
4 let
5 sshKeys = import ../ssh-keys.nix;
6 in
@@ -11,8 +11,11 @@ in
11 ../modules/prometheus
12 ../modules/tarball-mirror.nix
13 ../modules/wireguard.nix
14 + "${modulesPath}/virtualisation/amazon-image.nix"
15 ];
16
17 + ec2.hvm = true;
18 +
19 networking.hostName = "bastion";
20
21 system.configurationRevision = flakes.self.rev
@@ -58,14 +61,6 @@ in
61 AcceptEnv AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY FASTLY_API_KEY GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL
62 '';
63
61 - boot.loader.grub.enable = true;
62 - boot.loader.grub.device = "nodev";
63 -
64 - fileSystems."/" = {
65 - fsType = "ext4";
66 - device = "/dev/disk/by-label/nixos";
67 - };
68 -
64 fileSystems."/scratch" = {
65 autoFormat = true;
66 fsType = "ext4";