Browse Source

Publish NixOS LXC

main
Victor Roest 2 years ago
parent
commit
ebe62c2db2
Signed by: 0x76 GPG Key ID: A3923C699D1A3BDA
  1. 1
      .gitattributes
  2. BIN
      assets/images/nixos-proxmox-lxc-features.png
  3. BIN
      assets/images/nixos-rebuild-switch.png
  4. BIN
      assets/webm/upload-nixos-container.webm
  5. 117
      content/posts/nixos-proxmox-lxc.md
  6. 146
      content/posts/proxmox-nixos-lxc.md
  7. 2
      layouts/shortcodes/img.html
  8. 4
      layouts/shortcodes/webm.html

1
.gitattributes

@ -1 +1,2 @@
*.png filter=lfs diff=lfs merge=lfs -text
*.webm filter=lfs diff=lfs merge=lfs -text

BIN
assets/images/nixos-proxmox-lxc-features.png (Stored with Git LFS)

Binary file not shown.

BIN
assets/images/nixos-rebuild-switch.png (Stored with Git LFS)

Binary file not shown.

BIN
assets/webm/upload-nixos-container.webm (Stored with Git LFS)

Binary file not shown.

117
content/posts/nixos-proxmox-lxc.md

@ -0,0 +1,117 @@
---
title: "NixOS inside LXC"
date: 2021-02-21T18:56:48+01:00
draft: false
tags:
- Proxmox
- NixOS
categories:
- tutorial
---
This tutorial will explain how to install NixOS as an LXC container inside of proxmox.
Do be warned that this setup isn't extensively tested for stability or officially supported.
# Getting the container tarball
The first step is to download the NixOS container tarball. To do so
simply go to its [pipeline][nixos-tar]. And click on the latest successful build, and download the corresponding `.tar.xz` file.
{{< img path="images/nixos-buildproduct.png" alt="A screenshot of the NixOS container tarball download page" >}}
After it's downloaded we should rename the file to follow proxmox conventions (recommended but optional):
```sh
mv nixos-system-x86_64-linux.tar.xz nixos-$RELEASE-default_$BUILDID_amd64.tar.xz
```
## Uploading to proxmox
Uploading it to proxmox is quite easy just go to your storage,
most likely called "local" then to "CT Templates" and click on upload and upload the tarball.
{{< webm path="webm/upload-nixos-container.webm" >}}
# Creating the container
To create the container on proxmox we need to either `ssh` into it or use the web shell.
After in a shell on the proxmox host execute the following command. But, make sure you understand what the options
do before executing it. You can see the [proxmox docs][proxmox-lxc] if you are unsure.
```sh
pct create $(nextid) \
--arch amd64 \
--description nixos-template \
local:vztmpl/nixos-$RELEASE-default_$BUILDID_amd64.tar.xz \
--ostype unmanaged \
--net0 name=eth0 \
--storage local-lvm \
--unprivileged 1
```
after running this the container should show up in the Proxmox Web UI.
## Fix LXC config
After creating the container we need to make a simple edit to the lxc config file located in `/etc/pve/lxc/$ID.conf`,
`$ID` being the ID you passed in the previous step, if unsure you can check the web UI.
After opening the file add the following line to the bottom:
```conf
lxc.init.cmd: /init
```
This will point proxmox to the correct init binary of NixOS.
## Enable nesting
We are now done with all the cli configuration and need to enable a feature flag in the web interface.
Under container options, go to "Features" and enable "Nesting". This is needed as nix makes heavy use of sandboxing.
{{< img path="images/nixos-proxmox-lxc-features.png" alt="Picture showing the proxmox featues dialog with nesting enabled" >}}
_Do note the following (from the [proxmox wiki][proxmox-lxc])_
> Nesting is best used with unprivileged containers with additional id mapping. Note that this will expose procfs and sysfs contents of the host to the guest.
## Fix Network settings
If you didn't specify a full network configuration during container creation you must now
do so in the web UI or else the container won't start. The easiest being to just to set both ipv4 and ipv6 to dhcp.
Do mind that these network settings seem mostly to just be defaults that are able to be overwritten inside of the container,
except things like MAC address presumably.
## In-Container tweaks
Now you can finally start up the container! But we are not done yet, we need to set some minor settings
to make NixOS play nice with the fact that it is running inside of an lxc container.
### Populate Nixpkgs
`nixpkgs` isn't properly initialized when booting a fresh container, which would result in errors when running other
nix commands, to fix this simply run:
```bash
nix-channel --update
```
### configuration.nix tweak
Finally we need to add a small tweak inside of `/etc/nixos/configuration.nix`. We will
supress the `sys-kernel-debug.mount` systemd unit as it will otherwise error upon
every invocation of `nixos-rebuild switch` which can be annoying.
```nix
# Supress sys-kernel-debug
systemd.suppressedSystemUnits = [
"sys-kernel-debug.mount"
];
```
## Done!
After doing the final tweaks you should be done and able to run \
`nixos-rebuild switch` without any errors.
{{< img path="images/nixos-rebuild-switch.png" alt="A screenshot of a succesful `nixos-rebuild switch` invocation" >}}
# References
This guide is based heavily on the resources below
* [Proxmox LXC Docs][proxmox-lxc]
* [NixOS LXC Docs][nixos-lxc]
* [NixOS issue 1 on LXC][nixos-issue-1]
* [NixOS issue 2 on LXC][nixos-issue-2]
* [Nix Generate issue on LXC][nix-generate-issue]
[nixos-tar]: https://hydra.nixos.org/job/nixos/release-20.09/nixos.containerTarball.x86_64-linux
[nixos-lxc]: https://nixos.wiki/wiki/Proxmox_Linux_Container
[proxmox-lxc]: https://pve.proxmox.com/wiki/Linux_Container
[nixos-issue-1]: https://github.com/NixOS/nixpkgs/issues/9735
[nixos-issue-2]: https://github.com/NixOS/nixpkgs/issues/43781
[nix-generate-issue]: https://github.com/nix-community/nixos-generators/issues/41

146
content/posts/proxmox-nixos-lxc.md

@ -1,146 +0,0 @@
---
title: "NixOS inside LXC"
date: 2021-02-21T18:56:48+01:00
draft: true
tags:
- Proxmox
- NixOS
categories:
- tutorial
---
This tutorial will explain on how to install NixOS as an LXC container inside of proxmox.
Do be warned that this setup isn't entirely stable or supported.
# Getting the container tarball
Go to the pipeline for the [NixOS 20.09 Container Tarball][nixos-tar]
Then click on the latest successful build and download the corresponding `.tar.xz`.
{{< img path="images/nixos-buildproduct.png" >}}
After it's downloaded we should rename the file to follow proxmox conventions (recommended but optional):
```sh
mv nixos-system-x86_64-linux.tar.xz nixos-$RELEASE-default_$BUILDID_amd64.tar.xz
```
## Uploading to proxmox
Uploading it to proxmox is quite easy just go to your storage,
most likely called "local" then in "CT Templates" and click on upload and upload the tarball.
# Creating the container
To create the container on proxmox we need to either `ssh` into it or use the web shell.
After in a shell on the proxmox host execute the following command. But, make sure you understand what the options
do before executing it. You can see the [proxmox docs][proxmox-lxc] if you are unsure.
```sh
pct create $(nextid) \
--arch amd64 \
--description nixos-template \
local:vztmpl/nixos-$RELEASE-default_$BUILDID_amd64.tar.xz \
--ostype unmanaged \
--net0 name=eth0 \
--storage local-lvm \
--unprivileged 1
```
after running this the container should show up in the Proxmox Web UI.
## Fix LXC config
After creating the container we need to make a simple edit to the lxc config file located in `/etc/pve/lxc/$ID.conf`,
`$ID` being the ID you passed in the previous step, if unsure you can check the web UI.
After opening the file add the following line to the bottom:
```conf
lxc.init.cmd: /init
```
This will point proxmox to the correct init binary.
## Fix Network settings
If you didn't specify a full network configuration during container creation you must now
do so in the web UI or else the container won't start. The easiest being to just set ipv4 and ipv6 to dhcp.
## In-Container tweaks
Now you can finally start up the container! But we are not done yet, we need to set some minor settings
to make NixOS play nice with the fact that it is running inside of an lxc container.
### /proc fix
For some reason `/proc` is mounted with (to NixOS) unexpected permissions to fix this we need to
run the following:
```bash
mkdir -p /mnt/proc
mount -t proc proc /mnt/proc
```
### Populate Nixpkgs
`nixpkgs` isn't properly initialized when booting a fresh container, which would result in errors when running other
nix commands, to fix this simply run:
```bash
nix-channel --update
```
### configuration.nix tweaks
Finally we will add some tweaks inside of `/etc/nixos/configuration.nix`. The first one being simply making
the previously done `/proc` hack permanent, and the second one suppresses some annoying systemd service warnings.
```nix
# Make /proc hack permanent
fileSystems."/mnt/proc" = {
fsType = "proc";
device = "/proc";
};
# Supresses a bunch of systemd errors caused by running inside of LXC
systemd.suppressedSystemUnits = [
"console-getty.service"
"getty@.service"
"systemd-udev-trigger.service"
"systemd-udevd.service"
"sys-fs-fuse-connections.mount"
"sys-kernel-debug.mount"
"dev-mqueue.mount"
];
```
# Expected Errors
As this setup is fairly unconvential there are some errors that will occur when running nix,
especially `nixos-rebuild switch`. However as far as I can tell these don't seem to pose any real problem.
For example, these are the errors that I get
```shell
> nixos-rebuild switch
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...
mount: /dev: cannot remount devtmpfs read-write, is write-protected.
mount: /dev/pts: cannot remount devpts read-write, is write-protected.
mount: /dev/shm: cannot remount tmpfs read-write, is write-protected.
mount: /proc: cannot remount proc read-write, is write-protected.
mount: /run: cannot remount tmpfs read-write, is write-protected.
mount: /run/keys: cannot remount ramfs read-write, is write-protected.
mount: /run/wrappers: cannot remount tmpfs read-write, is write-protected.
Activation script snippet 'specialfs' failed (32)
reloading user units for root...
setting up tmpfiles
fchownat() of /run/keys failed: Read-only file system
fchownat() of /run/keys failed: Read-only file system
warning: error(s) occurred while switching to the new configuration
```
Some (or all) of these errors can be fixed by running a privileged and apparmor unconstrained container,
but it doesn't seem to have any real effect I wouldn't recommend doing this. There are some open issues
on the nix repos about these problems (linked below), but no real progress has been made it seems.
# References
This guide is based heavily on the resources below
* [Proxmox LXC Docs][proxmox-lxc]
* [NixOS LXC Docs][nixos-lxc]
* [NixOS issue 1 on LXC][nixos-issue-1]
* [NixOS issue 2 on LXC][nixos-issue-2]
* [Nix Generate issue on lxc][nix-generate-issue]
[nixos-tar]: https://hydra.nixos.org/job/nixos/release-20.09/nixos.containerTarball.x86_64-linux
[nixos-lxc]: https://nixos.wiki/wiki/Proxmox_Linux_Container
[proxmox-lxc]: https://pve.proxmox.com/wiki/Linux_Container
[nixos-issue-1]: https://github.com/NixOS/nixpkgs/issues/9735
[nixos-issue-2]: https://github.com/NixOS/nixpkgs/issues/43781
[nix-generate-issue]: https://github.com/nix-community/nixos-generators/issues/41

2
layouts/shortcodes/img.html

@ -1,2 +1,2 @@
{{ $image := resources.Get (.Get "path") }}
<img src="{{ $image.Permalink }}" alt="">
<img src="{{ $image.Permalink }}" alt="{{ (.Get "alt") }}">

4
layouts/shortcodes/webm.html

@ -0,0 +1,4 @@
{{ $webm := resources.Get (.Get "path") }}
<video controls autoplay loop width="100%">
<source src="{{ $webm.Permalink }}" type="video/webm">
</video>
Loading…
Cancel
Save