Fledged out NixOS LXC

.gitattributes

@@ -0,0 +1 @@
+*.png filter=lfs diff=lfs merge=lfs -text

config.yml

@@ -2,9 +2,17 @@ baseURL: https://blog.xirion.net/
 languageCode: en-gb
 title: Xirion.net Blog
 theme: PaperMod
-enableRobotsTXT: true
 copyright: "Source available on [Gitea](https://git.xirion.net/0x76/blog.xirion.net)"
 
+enableRobotsTXT: true
+buildDrafts: false
+buildFuture: false
+buildExpired: false
+
+minify:
+  disableXML: true
+  minifyOutput: true
+
 params:
   env: production
   description: "A blog about things"

content/posts/proxmox-nixos-lxc.md

@@ -1,5 +1,5 @@
 ---
-title: "Proxmox NixOS LXC"
+title: "NixOS inside LXC"
 date: 2021-02-21T18:56:48+01:00
 draft: true
 tags:
@@ -9,8 +9,138 @@ categories:
 - tutorial
 ---
 
-* set networking to dhcp inside of proxmox
-* https://github.com/NixOS/nixpkgs/issues/9735#issuecomment-559264857 (apparmor)
-* https://nixos.wiki/wiki/Proxmox_Linux_Container + unpriv
-* https://github.com/NixOS/nixpkgs/issues/9735#issuecomment-451477980 (proc)
-* https://pve.proxmox.com/wiki/Linux_Container
+This tutorial will go through on how to install NixOS as an LXC container inside of proxmox.
+
+## Getting the container tarball
+Go to the pipeline for the [NixOS 20.09 Container Tarball][nixos-tar]
+Then click on the latest successful build and download the corresponding `.tar.xz`.
+{{< img path="images/nixos-buildproduct.png" >}}
+
+After it's downloaded we should rename the file to follow proxmox conventions (recommended but optional):
+```sh
+mv nixos-system-x86_64-linux.tar.xz nixos-$RELEASE-default_$BUILDID_amd64.tar.xz
+```
+
+### Uploading to proxmox
+Uploading it to proxmox is quite easy just go to your storage, 
+most likely called "local" then in  "CT Templates" and click on upload and upload the tarball.
+
+## Creating the container
+To create the container on proxmox we need to either `ssh` into it or use the web shell.
+After in a shell on the proxmox host execute the following command. But, make sure you understand what the options
+do before executing it. You can see the [proxmox docs][proxmox-lxc] if you are unsure.
+
+```sh
+pct create $(nextid) \
+  --arch amd64 \
+  --description nixos-template \
+  local:vztmpl/nixos-$RELEASE-default_$BUILDID_amd64.tar.xz \
+  --ostype unmanaged \
+  --net0 name=eth0 \
+  --storage local-lvm \
+  --unprivileged 1
+```
+
+after running this the container should show up in the Proxmox Web UI.
+
+### Fix LXC config
+After creating the container we need to make a simple edit to the lxc config file located in `/etc/pve/lxc/$ID.conf`, 
+`$ID` being the ID you passed in the previous step, if unsure you can check the web UI.
+
+After opening the file add the following line to the bottom:
+```conf
+lxc.init.cmd: /init
+```
+This will point proxmox to the correct init binary.
+
+### Fix Network settings
+If you didn't specify a full network configuration during container creation you must now
+do so in the web UI or else the container won't start. The easiest being to just set ipv4 and ipv6 to dhcp.
+
+### In-Container tweaks
+Now you can finally start up the container! But we are not done yet, we need to set some minor settings
+to make NixOS play nice with the fact that it is running inside of an lxc container.
+
+#### /proc fix
+For some reason `/proc` is mounted with (to NixOS) unexpected permissions to fix this we need to 
+run the following:
+```bash
+mkdir -p /mnt/proc
+mount -t proc proc /mnt/proc
+```
+
+#### Populate Nixpkgs
+`nixpkgs` isn't properly initialized when booting a fresh container, which would result in errors when running other
+nix commands, to fix this simply run:
+```bash
+nix-channel --update
+```
+
+#### configuration.nix tweaks
+Finally we will add some tweaks inside of `/etc/nixos/configuration.nix`. The first one being simply making
+the previously done `/proc` hack permanent, and the second one suppresses some annoying systemd service warnings.
+```nix
+  # Make /proc hack permanent
+  fileSystems."/mnt/proc" = {
+        fsType = "proc";
+        device = "/proc";
+  };
+
+  # Supresses a bunch of systemd errors caused by running inside of LXC
+  systemd.suppressedSystemUnits = [
+    "console-getty.service"
+    "getty@.service"
+    "systemd-udev-trigger.service"
+    "systemd-udevd.service"
+    "sys-fs-fuse-connections.mount"
+    "sys-kernel-debug.mount"
+    "dev-mqueue.mount"
+  ];
+```
+
+## Expected Errors
+As this setup is fairly unconvential there are some errors that will occur when running nix,
+especially `nixos-rebuild switch`. However as far as I can tell these don't seem to pose any real problem.
+
+For example, these are the errors that I get
+```shell
+> nixos-rebuild switch
+building Nix...
+building the system configuration...
+activating the configuration...
+setting up /etc...
+mount: /dev: cannot remount devtmpfs read-write, is write-protected.
+mount: /dev/pts: cannot remount devpts read-write, is write-protected.
+mount: /dev/shm: cannot remount tmpfs read-write, is write-protected.
+mount: /proc: cannot remount proc read-write, is write-protected.
+mount: /run: cannot remount tmpfs read-write, is write-protected.
+mount: /run/keys: cannot remount ramfs read-write, is write-protected.
+mount: /run/wrappers: cannot remount tmpfs read-write, is write-protected.
+Activation script snippet 'specialfs' failed (32)
+reloading user units for root...
+setting up tmpfiles
+fchownat() of /run/keys failed: Read-only file system
+fchownat() of /run/keys failed: Read-only file system
+warning: error(s) occurred while switching to the new configuration
+nixos-rebuild switch  14.80s user 12.45s system 17% cpu 2:38.52 total
+```
+
+Some (or all) of these errors can be fixed by running a privileged and apparmor unconstrained container,
+but it doesn't seem to have any real effect I wouldn't recommend doing this. There are some open issues
+on the nix repos about these problems (linked below), but no real progress has been made it seems.
+
+# References
+This guide is based heavily on the resources below
+
+* [Proxmox LXC Docs][proxmox-lxc]
+* [NixOS LXC Docs][nixos-lxc]
+* [NixOS issue 1 on LXC][nixos-issue-1]
+* [NixOS issue 2 on LXC][nixos-issue-2]
+* [Nix Generate issue on lxc][nix-generate-issue]
+
+[nixos-tar]: https://hydra.nixos.org/job/nixos/release-20.09/nixos.containerTarball.x86_64-linux
+[nixos-lxc]: https://nixos.wiki/wiki/Proxmox_Linux_Container
+[proxmox-lxc]: https://pve.proxmox.com/wiki/Linux_Container
+[nixos-issue-1]: https://github.com/NixOS/nixpkgs/issues/9735
+[nixos-issue-2]: https://github.com/NixOS/nixpkgs/issues/43781
+[nix-generate-issue]: https://github.com/nix-community/nixos-generators/issues/41

layouts/shortcodes/img.html

@@ -0,0 +1,2 @@
+{{ $image := resources.Get (.Get "path") }}
+<img src="{{ $image.Permalink }}" alt="">