Installing NixOS with ZFS mirrored boot

2023-01-31T00:00:00+00:00

// TODO: add PlantUML diagrams</p>

Overview</h2>
In this post, we're going to set up a ZFS mirrored boot system with full-disk encryption that is unlockable remotely.</p>

Preparing the installation medium</h2>

This step may vary depending on what system you're going to install NixOS into.</p>

This post assumes that you're installing this on a normal server, with a minimal NixOS image.</p>

The community-maintained NixOS wiki</a> contains guides to install NixOS to devices in other conditions, such as a server with only remote access.</p>

You will need a USB stick before proceeding to the next step.</p>

First, download the latest NixOS image, and flash it:</p>

$ curl</span> -L</span> https://channels.nixos.org/nixos-unstable/latest-nixos-minimal-x86_64-linux.iso</span> -O</span> nixos.iso
</span>$ dd if=./nixos.iso of=/dev/sdX bs=1M status=progress
</span></code></pre>
If your target machine architecture is not x86_64</code>, replace it with your
desired architecture (e.g. i686</code>, aarch64</code>).</p>
After the image has been successfully flashed into your installation medium,
unplug it and boot using the medium on the target machine.</p>
Preparing Disks</h2>
We'll start by defining variables pointing to each disk by ID.</p>
According to the Archlinux.org Wiki</a>, If you create zpools using device names
(e.g. /dev/sda</code>), ZFS might not be able to detect zpools intermittently on
boot.</p>
You can grab the ID via ls -lh /dev/disk/by-id/</code>.</p>
DISK1</span>=</span>/dev/disk/by-id/ata-VENDOR-ID-OF-THE-FIRST-DRIVE
</span>DISK2</span>=</span>/dev/disk/by-id/ata-VENDOR-ID-OF-THE-SECOND-DRIVE
</span></code></pre>
Partitioning</h3>
Then we'll partition our disks. Since this is a mirrored setup, we'll have to do
the exactly same operation twice. Fortunately, bash function come into rescue.</p>
The partition structure is the following:</p>
1GiB Boot | ~Remaining ZFS
</span></code></pre>
partition</span>() {
</span>    sgdisk</span> --zap-all </span>"$1"
</span>    sgdisk</span> -n</span> 1:0:+1GiB</span> -t</span> 1:EF00</span> -c</span> 1:boot </span>"$1"
</span>    </span># Swap is omitted.
</span>    sgdisk</span> -n</span> 2:0:0</span> -t</span> 2:BF01</span> -c</span> 2:zfs </span>"$1"
</span>    sgdisk</span> --print </span>"$1"
</span>}
</span>
</span>partition $DISK1
</span>partition $DISK2
</span></code></pre>
Creating vfat filesystem for boot</h3>
Boot partitions should be formatted with 'vfat', in order for it to mount and
function without issues.</p>
mkfs.vfat $DISK1-part1
</span>mkfs.vfat $DISK2-part1
</span></code></pre>
Configuring ZFS pool</h3>
This dataset structure is based on Erase your darlings</a>.</p>
Now that we're done partitioning our disks, we'll create a ZFS pool named
'rpool', which is mirrored. This will prompt you to enter a passphrase for your
new ZFS pool.</p>
zpool create \
</span>    -o</span> ashift=12 \
</span>    -O</span> mountpoint=none</span> -O</span> atime=off</span> -O</span> acltype=posixacl</span> -O</span> xattr=sa \
</span>    -O</span> compression=lz4</span> -O</span> encryption=aes-256-gcm</span> -O</span> keyformat=passphrase \
</span>    rpool mirror \
</span>    $DISK1-part2 $DISK2-part2
</span></code></pre>
Then, we create a 'root dataset' which is / (root)</code> for the target machine,
then snapshot the empty state as 'blank'.</p>
zfs create</span> -p -o</span> mountpoint=legacy rpool/local/root
</span>zfs snapshot rpool/local/root@blank
</span></code></pre>
Note the 'local' after rpool. In this setup, 'local' is treated as unimportant
data, i.e. packages, root, etc., Whereas 'safe' is treated as important data,
which needs to be backed up.</p>
And mount it:</p>
mount</span> -t</span> zfs rpool/local/root /mnt
</span></code></pre>
Then we mount the multiple boot partitions we created:</p>
mkdir /mnt/boot
</span>mkdir /mnt/boot-fallback
</span>
</span>mount $DISK1-part1 /mnt/boot
</span>mount $DISK2-part1 /mnt/boot-fallback
</span></code></pre>
Create and mount a dataset for /nix</code>:</p>
zfs create</span> -p -o</span> mountpoint=legacy rpool/local/nix
</span>mkdir /mnt/nix
</span>mount</span> -t</span> zfs rpool/local/nix /mnt/nix
</span></code></pre>
And a dataset for /home</code>:</p>
zfs create</span> -p -o</span> mountpoint=legacy rpool/safe/home
</span>mkdir /mnt/home
</span>mount</span> -t</span> zfs rpool/safe/home /mnt/home
</span></code></pre>
And a dataset for states that needs to be persisted between boots:</p>
zfs create</span> -p -o</span> mountpoint=legacy rpool/safe/persist
</span>mkdir /mnt/persist
</span>mount</span> -t</span> zfs rpool/safe/persist /mnt/persist
</span></code></pre>
Note: All states will be wiped each boot after setting up
these</a>.
Make sure to put states that need to persist on /persist</code>.</p>
Configuring NixOS</h2>
Now that we're done with partitions and ZFS, it's time to declaratively
configure the machine. This step may vary depending on your machine,
please consult the docs when in doubt.</p>
Getting the base configuration</h3>
In this post, we're going to use plain nixos-generate-config</code> to get a base
configuration files for the machine.</p>
nixos-generate-config</span> --root</span> /mnt
</span></code></pre>
Erasing your darlings</h3>
In the previous step</a>, we've made a snapshot of blank
root to roll back to it each boot, to keep the system stateless.</p>
Add this to the configuration.nix</code> to wipe the root dataset on each boot by
rolling back to the blank snapshot after the devices are made available:</p>
{
</span>  </span>boot</span>.</span>initrd</span>.</span>postDeviceCommands </span>= </span>lib</span>.</span>mkAfter </span>''
</span>    zfs rollback -r rpool/local/root@blank
</span>  ''</span>;
</span>}
</span></code></pre>
Configuring Bootloader</h3>
In order to get ZFS to work, we need the following options to be set:</p>
{
</span>  </span>boot</span>.</span>supportedFilesystems </span>= </span>[ </span>"zfs" </span>];
</span>  </span>networking</span>.</span>hostId </span>= </span>"<8 random chars>"</span>;
</span>}
</span></code></pre>
You can grab your machine ID at /etc/machine-id</code> for the hostId</code>.</p>
Then we'll configure grub:</p>
{
</span>  </span># Whether installer can modify the EFI variables.
</span>  </span># If you encounter errors, set this to `false`.
</span>  </span>boot</span>.</span>loader</span>.</span>efi</span>.</span>canTouchEfiVariables </span>= </span>true</span>;
</span>
</span>  </span>boot</span>.</span>loader</span>.</span>grub</span>.</span>enable </span>= </span>true</span>;
</span>  </span>boot</span>.</span>loader</span>.</span>grub</span>.</span>efiSupport </span>= </span>true</span>;
</span>  </span>boot</span>.</span>loader</span>.</span>grub</span>.</span>device </span>= </span>"nodev"</span>;
</span>
</span>  </span># This should be done automatically, but explicitly declare it just in case.
</span>  </span>boot</span>.</span>loader</span>.</span>grub</span>.</span>copyKernels </span>= </span>true</span>;
</span>  </span># Make sure that you've listed all of the boot partitions here.
</span>  </span>boot</span>.</span>loader</span>.</span>grub</span>.</span>mirroredBoots </span>= </span>[
</span>    { </span>path </span>= </span>"/boot"</span>; </span>devices </span>= </span>[</span>"/dev/disk/by-uuid/<ID-HERE>"</span>]; }
</span>    { </span>path </span>= </span>"/boot-fallback"</span>; </span>devices </span>= </span>[</span>"/dev/disk/by-uuid/<ID-HERE>"</span>]; }
</span>    </span># ...
</span>  ];
</span>}
</span></code></pre>
Handling boot partitions gracefully</h3>
By default, NixOS will throw an error and complain about it when there is a
missing partition/disk. Since we want the server to boot smoothly even if there
is a missing boot partition, so we need to set the 'nofail' option to those
partitions:</p>
{
</span>  </span>fileSystems</span>.</span>"/boot"</span>.</span>options </span>= </span>[ </span>"nofail" </span>];
</span>  </span>fileSystems</span>.</span>"/boot-fallback"</span>.</span>options </span>= </span>[ </span>"nofail" </span>];
</span>}
</span></code></pre>
Enabling Remote ZFS Unlock</h3>
On each boot, ZFS will ask for a passphrase to unlock the ZFS pool.
To work around this issue, we can start an SSH server in initrd</code>, that is going
to live until the pool is unlocked.</p>
Note: If you rename the keys after, you may have some trouble rolling back to
previous generations: See here</a> for details.</p>
To achieve that, we'll first have to generate an SSH host key for the initrd:</p>
ssh-keygen</span> -t</span> ed25519</span> -N </span>""</span> -f</span> /mnt/boot/initrd-ssh-key
</span>
</span># Each boot partition should have the same key
</span>cp /mnt/boot/initrd-ssh-key /mnt/boot-fallback/initrd-ssh-key
</span></code></pre>
Then configure initrd</code>:</p>
{
</span>  </span>boot</span>.</span>kernelModules </span>= </span>[ </span>"<YOUR-NETWORK-CARD>" </span>];
</span>  </span>boot</span>.</span>initrd</span>.</span>kernelModules </span>= </span>[ </span>"<YOUR-NETWORK-CARD>" </span>];
</span>
</span>  </span># DHCP Configuration, comment on Static IP
</span>  </span>networking</span>.</span>networkmanager</span>.</span>enable </span>= </span>false</span>;
</span>  </span>networking</span>.</span>useDHCP </span>= </span>true</span>;
</span>
</span>  </span># Uncomment on Static IP
</span>  </span># boot.kernelParams = [
</span>  </span>#   # See <https:#www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt> for documentation.
</span>  </span>#   # ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns0-ip>:<dns1-ip>:<ntp0-ip>
</span>  </span>#   # The server ip refers to the NFS server -- not needed in this case.
</span>  </span>#   "ip=<YOUR-IPV4-ADDR>::<YOUR-IPV4-GATEWAY>:<YOUR-IPV4-NETMASK>:<YOUR-HOSTNAME>-initrd:<YOUR-NETWORK-INTERFACE>:off:<DNS-IP>"
</span>  </span># ];
</span>
</span>  </span>boot</span>.</span>initrd</span>.</span>network</span>.</span>enable </span>= </span>true</span>;
</span>  </span>boot</span>.</span>initrd</span>.</span>network</span>.</span>ssh </span>= </span>{
</span>    </span>enable </span>= </span>true</span>;
</span>
</span>    </span># Using the same port as the actual SSH will cause clients to throw errors
</span>    </span># related to host key mismatch.
</span>    </span>port </span>= </span>2222</span>;
</span>
</span>    </span># This takes 'path's, not 'string's.
</span>    </span>hostKeys </span>= </span>[
</span>      </span>/boot/initrd-ssh-key
</span>      </span>/boot-fallback/initrd-ssh-key
</span>      </span># ...
</span>    ];
</span>
</span>    </span># Public ssh key to log into the initrd ssh
</span>    </span>authorizedKeys </span>= </span>[ </span>"<YOUR-SSH-PUBKEY>" </span>];
</span>  };
</span>  </span>boot</span>.</span>initrd</span>.</span>network</span>.</span>postCommands </span>= </span>''
</span>    cat <<EOF > /root/.profile
</span>    if pgrep -x "zfs" > /dev/null
</span>    then
</span>      zfs load-key -a
</span>      killall zfs
</span>    else
</span>      echo "ZFS is not running -- this could be a sign of failure."
</span>    fi
</span>    EOF
</span>  ''</span>;
</span>}
</span></code></pre>
Installing NixOS</h2>
Run nixos-install</code>, then reboot your machine.</p>
Note: Make sure that you've configured SSH and network for your machine,
failure to do so may result in an inaccessible system.</p>
That's it! Enjoy your fresh NixOS machine!</p>
Troubleshooting</h2>
Failed to import pool - more than one matching pool</h3>
This error might occur when</p>

one of your disks were previously used in another ZFS pool, and its metadata
weren't properly removed</li>
you messed up during install, and you repartitioning the disk without removing
its ZFS metadata.</li>
</ul>
This is because the ZFS metadata doesn't live on a partition, but on a disk.</p>
Note: the following operations will irrevocably delete ANY data on your disk!</p>
To remove those left behind:</p>
sgdisk</span> --zap-all </span>$DISK
</span># Overwrite first 256M of the disk, removing metadata
</span># In some cases just `wipefs -a` works, but I found this to be the most
</span># reliable way to wipe them no matter what operations were performed on the disk
</span># before.
</span>dd if=/dev/urandom bs=1M count=256 of=$DISK
</span></code></pre>
And then you can try the installation again.</p>
Conclusion</h2>
Acknowledgements</h2>
I wrote this article because I've noticed that I always forget some steps
during NixOS installation to a newly acquired server.</p>
I've compiled resources listed below to make a step-by-step guide for a setup I
find 'optimal'. Please do check out those resources!</p>

NixOS Discourse Thread</a></li>
Erase your darlings</a></li>
Remote, encrypted ZFS storage server with NixOS</a></li>
Encrypted ZFS mirror with mirrored boot on NixOS</a></li>
</ul>

Installing NixOS</h2>
Run `nixos-install</code>, then reboot your machine.</p>`
`Note: Make sure that you've configured SSH and network for your machine, failure to do so may result in an inaccessible system.</p>`
`That's it! Enjoy your fresh NixOS machine!</p>`

Conclusion</h2>

セピー

Installing NixOS with ZFS mirrored boot

Overview</h2>
In this post, we're going to set up a ZFS mirrored boot system with full-disk encryption that is unlockable remotely.</p>

Hello, world!

セピー

Installing NixOS with ZFS mirrored boot

Overview</h2> In this post, we're going to set up a ZFS mirrored boot system with full-disk encryption that is unlockable remotely.</p>

Partitioning</h3> Then we'll partition our disks. Since this is a mirrored setup, we'll have to do the exactly same operation twice. Fortunately, bash function come into rescue.</p> The partition structure is the following:</p>

Creating vfat filesystem for boot</h3> Boot partitions should be formatted with 'vfat', in order for it to mount and function without issues.</p>

Getting the base configuration</h3> In this post, we're going to use plain nixos-generate-config</code> to get a base configuration files for the machine.</p>

Configuring Bootloader</h3> In order to get ZFS to work, we need the following options to be set:</p>

Handling boot partitions gracefully</h3> By default, NixOS will throw an error and complain about it when there is a missing partition/disk. Since we want the server to boot smoothly even if there is a missing boot partition, so we need to set the 'nofail' option to those partitions:</p>

Enabling Remote ZFS Unlock</h3> On each boot, ZFS will ask for a passphrase to unlock the ZFS pool. To work around this issue, we can start an SSH server in initrd</code>, that is going to live until the pool is unlocked.</p>

Installing NixOS</h2> Run nixos-install</code>, then reboot your machine.</p> Note: Make sure that you've configured SSH and network for your machine, failure to do so may result in an inaccessible system.</p> That's it! Enjoy your fresh NixOS machine!</p>

Troubleshooting</h2> Failed to import pool - more than one matching pool</h3> This error might occur when</p>

Failed to import pool - more than one matching pool</h3> This error might occur when</p>

Conclusion</h2>

Acknowledgements</h2> I wrote this article because I've noticed that I always forget some steps during NixOS installation to a newly acquired server.</p> I've compiled resources listed below to make a step-by-step guide for a setup I find 'optimal'. Please do check out those resources!</p>

Hello, world!

Overview</h2>
In this post, we're going to set up a ZFS mirrored boot system with full-disk encryption that is unlockable remotely.</p>

Installing NixOS</h2>
Run `nixos-install</code>, then reboot your machine.</p>`
`Note: Make sure that you've configured SSH and network for your machine, failure to do so may result in an inaccessible system.</p>`
`That's it! Enjoy your fresh NixOS machine!</p>`