nix-builder bootstrap
The host configuration can be built before its cache signing secret exists.
Harmonia's socket remains stopped until SOPS installs the signing key at
/run/secrets/harmonia/signing-key.
Proxmox storage layout
The host configuration expects three filesystems. Keep the build scratch space separate from the store so a large build cannot fill the root filesystem.
| Mount point | Suggested size | Contents |
|---|---|---|
/ |
48 GiB | NixOS and mutable system state |
/var/lib/nix-build |
192 GiB | Disposable build scratch space |
/nix/store |
1 TiB | Fleet closures and binary cache |
The build-server policy starts emergency store GC below 64 GiB free and aims
for 128 GiB free. Persistent roots under /var/lib/nix-fleet/roots protect the
latest fleet builds from that GC. It also limits Nix to two concurrent
derivations while allowing each derivation to use every vCPU assigned to the
VM.
For the two dedicated ext4 data filesystems, remove the default root-reserved blocks once after formatting; keep the root filesystem's reserve intact:
sudo tune2fs -m 0 /dev/disk/by-label/nix-build
sudo tune2fs -m 0 /dev/disk/by-label/nix-store
Initial deployment
Once the VM is reachable as moons@nix-builder, deploy it from the repository:
nix run .#deploy -- .#nix-builder
deploy-rs uses the target's ssh-ng store, so the system closure is built on
the builder rather than copied from the laptop. Automatic and magic rollback
remain enabled.
Add the host SOPS recipient
After the VM has a stable SSH host key, derive its age recipient:
ssh-keyscan -t ed25519 nix-builder 2>/dev/null | ssh-to-age
Add the recipient to .sops.yaml and add a creation rule for
secrets/hosts/nix-builder/*.yaml. The admin YubiKey recipient should remain in
the same key group for recovery.
Generate the cache signing key
Run this on a trusted Nix machine, preferably with the temporary files on a tmpfs:
nix-store --generate-binary-cache-key \
cache.app.homelabs.run-1 \
harmonia.private \
harmonia.public
Create secrets/hosts/nix-builder/system.yaml with SOPS and store the complete
contents of harmonia.private at harmonia.signing-key:
harmonia:
signing-key: cache.app.homelabs.run-1:REDACTED
Copy the complete contents of harmonia.public to
modules/systems/nix/homelab-cache/public-key. The private plaintext file must
not be committed or retained.
After committing both encrypted/public files, select
networking.homelab-cache-client on each client host.
Redeploy the builder and verify the cache after installing the secret:
nix run .#deploy -- .#nix-builder
curl --fail http://nix-builder:5000/nix-cache-info
Normal operation
Run fleet-build on the builder to build and root every NixOS host, or pass a
list of host names to build only those hosts. Run fleet-deploy with the normal
deploy-rs target syntax when additional fleet nodes have been added to
flake/deploy.nix:
fleet-build
fleet-build x1g13 galleria
fleet-deploy .#nix-builder