Bastion: rescue!
When moving away from NixOps, it looks like we missed importing the amazon-image.nix, and the bootloader config which was added to the configuration.nix was a bit wrong, pointing grub to `nodev` when it should have been `/dev/xvda1`. Rescuing this image involved: 1. Snapshotted the disk 1. Shut down the (broken) bastion 1. Booted another NixOS machine 1. Detached /dev/sda1 (note: the sda*1* here is not a typo!) from within the EC2 Console 1. Attached the root disk to the new machine as /dev/sdf, which appeared to as /dev/nvme1n1 Then, `lsblk` and `mount` showed the NEW DISK being mounted as the root and /nix/store, but these tools were *lying*. Both disks had their root partition labeled "nixos". To fix this, we: 1. Detached the new disk from the rescue machine 1. Edited the rescue machine's /etc/nixos/configuration.nix to include: fileSystems."/".device = lib.mkForce "/dev/disk/by-label/rescue"; 1. Ran `nixos-rebuild boot` 1. Updated the rescue machine's root filesystem label to `e2label /dev/nvme0n1p2 rescue` 1. Rebooted and confirmed the root filesystem was mounted from by-label/rescue. Now, we re-stared the rescue process. 1. Attach the old machine's disk to the rescue machine 1. Mounted /dev/nmvme1n1p1 to /mnt 1. Made the changes in this commit, *EXCEPT* we also made the following change: boot.loader.grub.device = lib.mkForce "/dev/nvme1n1"; Note the `nvme1n1` was correct *for the rescue environment* but is not correct in any other environment. After rescuing the image and getting it booted, we deleted that line. We considered using /dev/disk/by-id instead, which probably would have been a good choice. We didn't use it just because we didn't. 1. nix-shell -p git nixUnstable 1. Inside this shell we ran `nixos-install --flake .#bastion` from `/mnt/home/deploy/src/nixos-org-configurations/bastion`. 1. Then we unmounted /mnt, detached the volume, and reattached the volume to the old machine and it booted correctly.