What a treat it is to review the Framework laptop a few months before I’ll be buying my own.

The Framework promises to be an powerful, high-end ultra thin laptop with the stand-out feature of easy repairs and upgrades. I think they’ve done it.

Let’s install NixOS.

Live Media

You’ll need an install image with Linux 5.13 or newer to have a working Wi-Fi card.

This presents a challenge: the ISOs provided by https://nixos.org today use Linux 5.10.

If you’re comfortable installing NixOS without a GUI, you can fetch a minimal 21.05 ISO with the latest kernel from Hydra.

Optionally, Building Our Own Live Media

Building your own customized install media is straightforward. You’ll need a Linux machine with Nix.

First, write the following in to a file named custom-media.nix:

{ pkgs, modulesPath, ... }: {
    imports = [
        "${modulesPath}/installer/cd-dvd/installation-cd-graphical-gnome.nix"
    ];

    boot.kernelPackages = pkgs.linuxPackages_latest;
}

Then enter a nix-shell with nixos-generators and build the media:

$ nix-shell -p nixos-generators
nix-shell$ nixos-generate -I nixpkgs=channel:nixos-unstable --format iso --configuration ./custom-media.nix 
unpacking 'https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz'...
these derivations will be built:
  ...snip...
/nix/store/gnnbjvd916yh1f4svbgrssq94550pbxl-nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso/iso/nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso

Then copy the ISO to my USB disk, which is called /dev/sda.

Note: the sync at the end is critical.

$ sudo cp /nix/store/gnnbjvd916yh1f4svbgrssq94550pbxl-nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso/iso/nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso /dev/sda
$ sudo sync

Disabling Secure Boot

1. Reboot
2. Enter the Firmware Configuration (Fn + F2)
3. Navigate to the Security tab
4. Select Secure Boot
5. Select Enforce Secure Boot
6. Select Disabled
7. Save and reboot with Fn + F10

Note: Secure Boot will prevent unsigned software from running. NixOS does not support Secure Boot today.

Booting Your Media

It appears the machine can boot from any USB port.

1. Reboot
2. Enter the Boot Manager (Fn + F12)
3. Select your USB disk

Installing NixOS

Follow the standard NixOS installation instructions.

After running nixos-generate-config, edit /mnt/etc/nixos/configuration.nix and add a few lines:

• boot.kernelPackages = pkgs.linuxPackages_latest; - for WiFi support
• services.fprintd.enable = true; for fingerprint support

Continue the installation procedure.

Seriously. That’s it.

Hardware Review

The review unit they shipped me is pre-release hardware, but well specced out and lovely:

• 11th Gen Intel i7-1185G7, up to 4.8GHz
• 16G of RAM
• Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
• 2 USB-C ports, a mini-display port, and a Micro SD slot

Its performance seems quite good, easily good enough for my day to day work. It has no problems driving my 38” ultrawide display.

The machine is silent except when I’m compiling, and it has never felt hot. After a long compilation some areas near the hinge are noticably warm, but the hand rests are still cool.

The keyboard and trackpad feel, to me, perfect. The keys feel crisp but not heavy. The trackpad feels large and precise. That said, click is a bit too clicky.

Hardware switches for the audio and camera are nice, but a bit hard to toggle. The switches are very small and smooth, making it difficult to move. The switches are part of the replacable bezel, and this may be easily fixed.

The power brick is UL Energy Verified and just slightly larger than the Dell XPS 9300 brick.

The Intel Wi-Fi 6 AX210 has integration issues with Linux. Sometimes the wifi firmware crashes on startup, but it seems to be no more frequent than 50/50. Once it is booted, the wifi is rock solid.

QR Codes on Everything

Everything has a QR code as a link to instructions. And I mean everything: down to the cable connecting the wall to the power brick.

However, many of them don’t actually lead anywhere. I wasn’t too surprised to see the power cable’s QR code was 404ing, but a bit surprised to see the link for RAM replacement 404ing. I am sure this is a work in progress and these QR codes will all work.

Relative to the XPS 9300

I want the smallest, lightest laptop I can buy, and the Framework is a strong contender here.

This laptop is essentially the same width and thickness: within a milimeter or two. The framework comes in at just 60 grams heavier than the XPS (1330g vs. 1270g.)

One major improvement over the 9300: The fingerprint sensor is supported by recent versions of fprintd out of the box, and no TOD patches are required. The sensor is fast and reliable, whereas the Dell XPS 9300’s TOD Goodix drivers are nearly worthless on Linux.

Some aspects of the Framework are a bit more “industrial” feeling:

• The cable from the brick to the wall looks like a desktop PC’s cable. Thick, shiny plastic. compared to the more polished, polished, on-brand look of the Dell cable.
• The rubber feet are chunky, and not super sticky.
• Opening the lid is not super easy like on the XPS.

Ready to Buy

Overall, I am extremely satisfied by Framework. The team seems great, and their values appear to match my own. I’ll be pulling out my wallet later this year when my current laptop isn’t so new. I think they’ve fulfilled their promise, and have created a refreshing machine: an ultralight which is servicable.

Yes, please.

The Nix community is about 18 years old now. Until recently (~6 years ago) the community was quite small. The project is now much bigger, and growing. The result is that, organizationally, we’re a bit immature. The RFC process is new, and only a few major changes have gone through it.

Unfortunately, Flakes was one of the very first major Nix features to go through the community’s relatively young RFC process.

As a community, we’re not accustomed to and practiced at breaking down large and fundamental changes to the ecosystem and shepherding them through RFCs. But there we were: we had a shiny new process and by golly, a change that deserves an RFC! We hadn’t even tied our shoes and yet we were attempting our first triathalon.

I believe everybody approached that RFC with really good intentions and hope that it would go well and be a productive process. In a lot of ways, it was productive. But the end of that RFC was not good.

What was one RFC probably should have been very theoretical “this is an idea, should we explore it?” followed by several RFCs about specific subsections and details about how Flakes would work.

But… we were so new to RFCs, so new to being a large project, we didn’t see the problems coming.

As a result, the RFC was closed, agreeing to make Flakes “experimental” but merge it into Nix anyway. Maybe in the future, a new RFC would be submitted to review the code as existed in Nix already.

This has caused quite a lot of bad feelings between all sorts of people. A lot of assumptions about motivations that I don’t believe hold up. Probably a good bit of distrust in to RFCs and the legitimacy of the process.

I find this so unfortunate. I believe Flakes are astonishingly important and will be very powerful, but the RFC experience has turned so many people against it on principle. Flakes have a lot of potential, and needs a strong community to work through their problems and help it flourish.

The damage to the perception of RFCs is real and tragic. I believe so deeply in this project and its ability to grow, and the organizational distrust from this makes that much, much more difficult.

I feel so disappointed in myself for not seeing the dangers of sending such a fundamental change to Nix the tool through a nearly brand new process which was a fundamental change to the Nix community. The process wasn’t ready for it, the participants weren’t ready for it.

I regret that so much.

I believe we can, as a project and community, move past it. It will take leadership and effort from a lot of people. Project leadership will have to make the first moves there, to mend the distrust and sow new seeds of cooperation. I know we can do this.

I have so much love for this project and its community. I feel so grateful to be part of it, surrounded every day by people so much smarter than me.

I hope to be part of the solution, to be part of the healing and growth.

I have been part of this project for just over five years now, and I am incredibly excited to be part of the next five.

The future of Nix is so bright I can hardly look right at it without looking down at less ambitious futures.

This was originally a series of tweets.

Over time, a system collects state on its root partition. This state lives in assorted directories like /etc and /var, and represents every under-documented or out-of-order step in bringing up the services.

“Right, run myapp-init.”

These small, inconsequential “oh, oops” steps are the pieces that get lost and don’t appear in your runbooks.

“Just download ca-certificates to … to fix …”

Each of these quick fixes leaves you doomed to repeat history in three years when you’re finally doing that dreaded RHEL 7 to RHEL 8 upgrade.

“Oh, touch /etc/ipsec.secrets or the l2tp tunnel won’t work.”

Immutable infrastructure gets us so close

Immutable infrastructure is a wonderfully effective method of eliminating so many of these forgotten steps. Leaning in to the pain by deleting and replacing your servers on a weekly or monthly basis means you are constantly testing and exercising your automation and runbooks.

The nugget here is the regular and indiscriminate removal of system state. Destroying the whole server doesn’t leave you much room to forget the little tweaks you made along the way.

These techniques work great when you meet two requirements:

• you can provision and destroy servers with an API call
• the servers aren’t inherently stateful

Long running servers

There are lots of cases in which immutable infrastructure doesn’t work, and the dirty secret is those servers need good tools the most.

Long-running servers cause long outages. Their runbooks are outdated and incomplete. They accrete tweaks and turn in to an ossified, brittle snowflake — except its arms are load-bearing.

Let’s bring the ideas of immutable infrastructure to these systems too. Whether this system is embedded in a stadium’s jumbotron, in a datacenter, or under your desk, we can keep the state under control.

FHS isn’t enough

The hard part about applying immutable techniques to long running servers is knowing exactly where your application state ends and the operating system, software, and configuration begin.

This is hard because legacy operating systems and the Filesystem Hierarchy Standard poorly separate these areas of concern. For example, /var/lib is for state information, but how much of this do you actually care about tracking? What did you configure in /etc on purpose?

The answer is probably not a lot.

You may not care, but all of this accumulation of junk is a tarpit. Everything becomes harder: replicating production, testing changes, undoing mistakes.

New computer smell

Getting a new computer is this moment of cleanliness. The keycaps don’t have oils on them, the screen is perfect, and the hard drive is fresh and unspoiled — for about an hour or so.

Let’s get back to that.

How is this possible?

NixOS can boot with only two directories: /boot, and /nix.

/nix contains read-only system configurations, which are specified by your configuration.nix and are built and tracked as system generations. These never change. Once the files are created in /nix, the only way to change the config’s contents is to build a new system configuration with the contents you want.

Any configuration or files created on the drive outside of /nix is state and cruft. We can lose everything outside of /nix and /boot and have a healthy system. My technique is to explicitly opt in and choose which state is important, and only keep that.

How this is possible comes down to the boot sequence.

For NixOS, the bootloader follows the same basic steps as a standard Linux distribution: the kernel starts with an initial ramdisk, and the initial ramdisk mounts the system disks.

And here is where the similarities end.

NixOS’s early startup

NixOS configures the bootloader to pass some extra information: a specific system configuration. This is the secret to NixOS’s bootloader rollbacks, and also the key to erasing our disk on each boot. The parameter is named systemConfig.

On every startup the very early boot stage knows what the system’s configuration should be: the entire system configuration is stored in the read-only /nix/store, and the directory passed through systemConfig has a reference to the config. Early boot then manipulates /etc and /run to match the chosen setup. Usually this involves swapping out a few symlinks.

If /etc simply doesn’t exist, however, early boot creates /etc and moves on like it were any other boot. It also creates /var, /dev, /home, and any other core directories that must be present.

Simply speaking, an empty / is not surprising to NixOS. In fact, the NixOS netboot, EC2, and installation media all start out this way.

Opting out

Before we can opt in to saving data, we must opt out of saving data by default. I do this by setting up my filesystem in a way that lets me easily and safely erase the unwanted data, while preserving the data I do want to keep.

My preferred method for this is using a ZFS dataset and rolling it back to a blank snapshot before it is mounted. A partition of any other filesystem would work just as well too, running mkfs at boot, or something similar. If you have a lot of RAM, you could skip the erase step and make / a tmpfs.

Opting out with ZFS

When installing NixOS, I partition my disk with two partitions, one for the boot partition, and another for a ZFS pool. Then I create and mount a few datasets.

My root dataset:

# zfs create -p -o mountpoint=legacy rpool/local/root

Before I even mount it, I create a snapshot while it is totally blank:

# zfs snapshot rpool/local/root@blank

And then mount it:

# mount -t zfs rpool/local/root /mnt

Then I mount the partition I created for the /boot:

# mkdir /mnt/boot
# mount /dev/the-boot-partition /mnt/boot

Create and mount a dataset for /nix:

# zfs create -p -o mountpoint=legacy rpool/local/nix
# mkdir /mnt/nix
# mount -t zfs rpool/local/nix /mnt/nix

And a dataset for /home:

# zfs create -p -o mountpoint=legacy rpool/safe/home
# mkdir /mnt/home
# mount -t zfs rpool/safe/home /mnt/home

And finally, a dataset explicitly for state I want to persist between boots:

# zfs create -p -o mountpoint=legacy rpool/safe/persist
# mkdir /mnt/persist
# mount -t zfs rpool/safe/persist /mnt/persist

Note: in my systems, datasets under rpool/local are never backed up, and datasets under rpool/safe are.

And now safely erasing the root dataset on each boot is very easy: after devices are made available, roll back to the blank snapshot:

{
  boot.initrd.postDeviceCommands = lib.mkAfter ''
    zfs rollback -r rpool/local/root@blank
  '';
}

I then finish the installation as normal. If all goes well, your next boot will start with an empty root partition but otherwise be configured exactly as you specified.

Opting in

Now that I’m keeping no state, it is time to specify what I do want to keep. My choices here are different based on the role of the system: a laptop has different state than a server.

Here are some different pieces of state and how I preserve them. These examples largely use reconfiguration or symlinks, but using ZFS datasets and mount points would work too.

Wireguard private keys

Create a directory under /persist for the key:

# mkdir -p /persist/etc/wireguard/

And use Nix’s wireguard module to generate the key there:

{
  networking.wireguard.interfaces.wg0 = {
    generatePrivateKeyFile = true;
    privateKeyFile = "/persist/etc/wireguard/wg0";
  };
}

NetworkManager connections

Create a directory under /persist, mirroring the /etc structure:

# mkdir -p /persist/etc/NetworkManager/system-connections

And use Nix’s etc module to set up the symlink:

{
  etc."NetworkManager/system-connections" = {
    source = "/persist/etc/NetworkManager/system-connections/";
  };
}

Bluetooth devices

Create a directory under /persist, mirroring the /var structure:

# mkdir -p /persist/var/lib/bluetooth

And then use systemd’s tmpfiles.d rules to create a symlink from /var/lib/bluetooth to my persisted directory:

{
  systemd.tmpfiles.rules = [
    "L /var/lib/bluetooth - - - - /persist/var/lib/bluetooth"
  ];
}

SSH host keys

Create a directory under /persist, mirroring the /etc structure:

# mkdir -p /persist/etc/ssh

And use Nix’s openssh module to create and use the keys in that directory:

{
  services.openssh = {
    enable = true;
    hostKeys = [
      {
        path = "/persist/ssh/ssh_host_ed25519_key";
        type = "ed25519";
      }
      {
        path = "/persist/ssh/ssh_host_rsa_key";
        type = "rsa";
        bits = 4096;
      }
    ];
  };
}

ACME certificates

Create a directory under /persist, mirroring the /var structure:

# mkdir -p /persist/var/lib/acme

And then use systemd’s tmpfiles.d rules to create a symlink from /var/lib/acme to my persisted directory:

{
  systemd.tmpfiles.rules = [
    "L /var/lib/acme - - - - /persist/var/lib/acme"
  ];
}

Answering the question “what am I about to lose?”

I found this process a bit scary for the first few weeks: was I losing important data each reboot? No, I wasn’t.

If you’re worried and want to know what state you’ll lose on the next boot, you can list the files on your root filesystem and see if you’re missing something important:

# tree -x /
├── bin
│   └── sh -> /nix/store/97zzcs494vn5k2yw-dash-0.5.10.2/bin/dash
├── boot
├── dev
├── etc
│   ├── asound.conf -> /etc/static/asound.conf
... snip ...

ZFS can give you a similar answer:

# zfs diff rpool/local/root@blank
M	/
+	/nix
+	/etc
+	/root
+	/var/lib/is-nix-channel-up-to-date
+	/etc/pki/fwupd
+	/etc/pki/fwupd-metadata
... snip ...

Your stateless future

You may bump in to new state you meant to be preserving. When I’m adding new services, I think about the state it is writing and whether I care about it or not. If I care, I find a way to redirect its state to /persist.

Take care to reboot these machines on a somewhat regular basis. It will keep things agile, proving your system state is tracked correctly.

This technique has given me the “new computer smell” on every boot without the datacenter full of hardware, and even on systems that do carry important state. I have deployed this strategy to systems in the large and small: build farm servers, database servers, my NAS and home server, my raspberry pi garage door opener, and laptops.

NixOS enables powerful new deployment models in so many ways, allowing for systems of all shapes and sizes to be managed properly and consistently. I think this model of ephemeral roots is yet another example of this flexibility and power. I would like to see this partitioning scheme become a reference architecture and take us out of this eternal tarpit of legacy.

NixOS’s filesystem architecture does cleanly separate user data from system data, and has a much easier job to do.

Traditional Linuxes

Because FHS mixes these two concerns across the entire hierarchy, splitting these concerns requires identifying every point across dozens of directories where the data is the system’s or the user’s. When adding ZFS to the mix, the installers typically have to create over a dozen datasets to accomplish this.

For example, Ubuntu’s upcoming ZFS support creates 16 datasets:

rpool/
├── ROOT
│   └── ubuntu_lwmk7c
│       ├── log
│       ├── mail
│       ├── snap
│       ├── spool
│       ├── srv
│       ├── usr
│       │   └── local
│       ├── var
│       │   ├── games
│       │   └── lib
│       │       ├── AccountServices
│       │       ├── apt
│       │       ├── dpkg
│       │       └── NetworkManager
│       └── www
└── USERDATA

Going through the great pains of separating this data comes with significant advantages: a recursive snapshot at any point in the tree will create an atomic, point-in-time snapshot of every dataset below.

This means in order to create a consistent snapshot of the system data, an administrator would only need to take a recursive snapshot at ROOT. The same is true for user data: take a recursive snapshot of USERDATA and all user data is saved.

NixOS

Because Nix stores all of its build products in /nix/store, NixOS doesn’t mingle these two concerns. NixOS’s runtime system, installed packages, and rollback targets are all stored in /nix.

User data is not.

This removes the entire complicated tree of datasets to facilitate FHS, and leaves us with only a few needed datasets.

Datasets

Design for the atomic, recursive snapshots when laying out the datasets.

In particular, I don’t back up the /nix directory. This entire directory can always be rebuilt later from the system’s configuration.nix, and isn’t worth the space.

One way to model this might be splitting up the data into three top-level datasets:

tank/
├── local
│   └── nix
├── system
│   └── root
└── user
    └── home

In tank/local, I would store datasets that should almost never be snapshotted or backed up. tank/system would store data that I would want periodic snapshots for. Most importantly, tank/user would contain data I want regular snapshots and backups for, with a long retention policy.

From here, you could add a ZFS dataset per user:

tank/
├── local
│   └── nix
├── system
│   └── root
└── user
    └── home
        ├── grahamc
        └── gustav

Or a separate dataset for /var:

tank/
├── local
│   └── nix
├── system
│   ├── var
│   └── root
└── user

Importantly, this gives you three buckets for independent and regular snapshots.

The important part is having /nix under its own top-level dataset. This makes it a “cousin” to the data you do want backup coverage on, making it easier to take deep, recursive snapshots atomically.

Properties

• Enable compression with compression=on. Specifying on instead of lz4 or another specific algorithm will always pick the best available compression algorithm.
• The dataset containing journald’s logs (where /var lives) should have xattr=sa and acltype=posixacl set to allow regular users to read their journal.
• Nix doesn’t use atime, so atime=off on the /nix dataset is fine.
• NixOS requires (as of 2020-04-11) mountpoint=legacy for all datasets. NixOS does not yet have tooling to require implicitly created ZFS mounts to settle before booting, and mountpoint=legacy plus explicit mount points in hardware-configuration.nix will ensure all your datasets are mounted at the right time.

I don’t know how to pick ashift, and usually just allow ZFS to guess on my behalf.

Partitioning

I only create two partitions:

1. /boot formatted vfat for EFI, or ext4 for BIOS
2. The ZFS dataset partition.

There are spooky articles saying only give ZFS entire disks. The truth is, you shouldn’t split a disk into two active partitions. Splitting the disk this way is just fine, since /boot is rarely read or written.

Note: If you do partition the disk, make sure you set the disk’s scheduler to none. ZFS takes this step automatically if it does control the entire disk.

On NixOS, you an set your scheduler to none via:

{ boot.kernelParams = [ "elevator=none" ]; }

Clean isolation

NixOS’s clean separation of concerns reduces the amount of complexity we need to track when considering and planning our datasets. This gives us flexibility later, and enables some superpowers like erasing my computer on every boot, which I’ll write about on Monday.

At the same time, the value of the container ecosystem is huge. Tying in to the schedulers, orchestration, and monitoring is very valuable.

Nix has been able to generate Docker images for several years now, however the typical approach to layering with Nix is to generate one fat image with all of the dependencies. This fat image offers no sharing, is slow to build, upload, and download.

In this post I talk about how I fix this problem and use Nix to automatically create multi-layered Docker images, allowing a high amount of caching between images.

Docker uses layers

Docker’s use of layering is well known, and its benefits are undeniable: sharing a “base” system is a simple abstraction which allows extending a well known image with your own code.

A Docker image is a sequence of layers, where each member is a filesystem diff, adding and removing files from its parent member:

Efficient layering is hard because there are no rules

When there are no restrictions on what a command will do, the only way to fully capture its effects is to snapshot the full filesystem.

Most package managers will write files to shared global directories like /usr, /bin, and /etc.

This means that the only way to represent the changes between installing package A and installing package B is to take a full snapshot of the filesystem.

As a user you might manually create rules to improve the behavior of the cache: add your code towards the end of a Dockerfile, or install common libraries in a single RUN instruction, even if you don’t want them all.

These rules make sense: If a Dockerfile adds code and then installs packages, Docker can’t cache the installation because it can’t know that the package installation isn’t influenced by the code addition. Docker also can’t know that installing package A has nothing to do with package B and the changes are separately cachable.

With restrictions, we can make better optimisations

Nix does have rules.

The most important and relevant rule when considering distribution and Docker layers is:

A package build can’t write to arbitrary places on the disk.

A build can only write to a specific directory known as $out, like /nix/store/ibfx7ryqnqf01qfzj4v7qhzhkd2v9mm7-file-5.34. When you add a new package to your system, you know it didn’t modify /etc or /bin.

How does file find its dependencies? It doesn’t – they are hard-coded:

$ ldd /nix/store/ibfx7ryqnqf01qfzj4v7qhzhkd2v9mm7-file-5.34/bin/file
	linux-vdso.so.1
	/nix/store/ibfx7ryqnqf01qfzj4v7qhzhkd2v9mm7-file-5.34/lib/libmagic.so.1
	/nix/store/bv6znzsv2qkbcwwa251dx7n5dshz3nr3-zlib-1.2.11/lib/libz.so.1
	/nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libc.so.6
	/nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ld-linux-x86-64.so.2

This provides great, cache-friendly properties:

1. You know exactly what path changed when you added file.
2. You know exactly what paths file depends on.
3. Once a path is created, it will never change again.

Think graphs, not layers

If you consider the properties Nix provides, you can see it already constructs a graph internally to represent software and its dependencies: it natively has a better understanding of the software than Docker is interested in.

Specifically, Nix uses a Directed Acyclic Graph to store build output, where each node is a specific, unique, and immutable path in /nix/store:

Or to use a real example, Nix itself can render a graph of a package’s dependencies:

Flattening Graphs to Layers

In a naive world we can simply walk the tree and create a layer out of each path:

and this image is valid: if you pulled any of these layers, you would automatically get all the layers below it, resulting in a complete set of dependencies.

Things get a bit more complicated for a graph with a wider graph, how do you flatten something like Bash:

If we had to flatten this to an ordered sequence, obviously bash-interactive-4.4-p23 is at the top, but does readline-7.0p5 come next? Why not bash-4.4p23?

It turns out we don’t have to solve this problem exactly, because I lied about how Docker represents layers.

How Docker really represents an Image

Docker’s layers are content addressable and aren’t required to explicitly reference a parent layer. This means a layer for readline-7.0p5 doesn’t have to mention that it has any relationship to ncurses-6.1 or glibc-2.27 at all.

Instead each image has a manifest which defines the order:

{
  "Layers": [
    "bash-interactive-4.4-p23",
    "bash-4.4p23",
    "readline-7.0p5",
     ...
  ]
}

If you have only built Docker images using a Dockerfile, then you would expect the way we flatten our graph to be critically important. If we sometimes picked readline-7.0p5 to come first and other times picked bash-4.4p23 then we may never make cache hits.

However since the Image defines the order, we don’t have to solve this impossible problem: we can order the layers in any way we want and the layer cache will always hit.

Docker doesn’t support an infinite number of layers

Docker has a limit of 125 layers, but big packages with lots of dependencies can easily have more than 125 store paths.

It is important that we still successfully build an image if we go over this limit, but what do we do with the extra layers?

In the interest of shortness, let’s pretend Docker only lets you have four layers, and we want to fit five. Out of the Bash example, which two paths do we combine in to one layer?

• bash-interactive-4.4-p23
• bash-4.4p23
• readline-7.0p5
• ncurses-6.1
• glibc-2.27

Smushing Layers

I decided the best solution is to combine the layers which are less likely to be a cache hit with other software. Picking the most low level, fundamental paths and making them a separate layer means my next image will most likely also share some of those layers too.

Ideally it would end up with at least glibc, and ncurses in separate layers. Visually, it is hard to tell if either readline or bash-4.4p23 would be better served as an individual layer. One of them should be, certainly.

My actual solution

My prioritization algorithm is a simple graph-based popularity contest. The idea is to weight each node more heavily the deeper and more references they have.

Starting with the dependency graph of Bash from before,

we first duplicate nodes in the graph so each node is only pointed to once:

we then replace each leaf node with a counter, starting at 1:

each node whose only children are counters are then combined with their children, and their children’s counters summed, then incremented:

we then repeat the process:

we repeat this process until there is only one node:

and finally we sort the paths in each popularity bucket by name to ensure the list is consistently generated to get the paths ordered by cachability:

• glibc-2.27: 10
• ncurses-6.1: 4
• bash-4.4-p23: 2
• readline-7.0p5: 2
• bash-interactive-4.4-p23: 1

This solution has properly put foundational paths which are most commonly referred to at the top, improving its chances of cache hit. The algorithm has also put the likely-to-change application right at the bottom in case the last layers need to be combined.

Let’s consider a much larger image. In this image, we set the maximum number of layers to 120, but the image has 200 store paths. Under this design the 119 most fundamental store paths will have their own layers, and we store the remaining 81 paths together in the 120th layer.

With this new approach of automatically layering store paths I can now generate images with very efficient caching between different images.

For a practical example of a PHP application with a MySQL database.

First we build a MySQL image:

# mysql.nix
let
  pkgs = import (builtins.fetchTarball {
    url = "https://github.com/grahamc/nixpkgs/archive/layered-docker-images.tar.gz";
    sha256 = "05a3jjcqvcrylyy8gc79hlcp9ik9ljdbwf78hymi5b12zj2vyfh6";
  }) {};
in pkgs.dockerTools.buildLayeredImage {
  name = "mysql";
  tag = "latest";
  config.Cmd = [ "${pkgs.mysql}/bin/mysqld" ];
  maxLayers = 120;
}

$ nix-build ./mysql.nix
$ docker load < ./result

Then we build a PHP image:

# php.nix
let
  pkgs = import (builtins.fetchTarball {
    url = "https://github.com/grahamc/nixpkgs/archive/layered-docker-images.tar.gz";
    sha256 = "05a3jjcqvcrylyy8gc79hlcp9ik9ljdbwf78hymi5b12zj2vyfh6";
  }) {};
in pkgs.dockerTools.buildLayeredImage {
  name = "grahamc/php";
  tag = "latest";
  config.Cmd = [ "${pkgs.php}/bin/php" ];
  maxLayers = 120;
}

$ nix-build ./php.nix
$ docker load < ./result

and export the two image layers:

$ docker inspect mysql | jq -r '.[] | .RootFS.Layers | .[]' | sort > mysql
$ docker inspect php | jq -r '.[] | .RootFS.Layers | .[]' | sort > php

and look at this, the PHP and MySQL images share twenty layers:

$ comm -1 -2 php mysql
sha256:0296e7b7d4b7d593fc533f06c5cc22f56c99c8ab0ed4301a6c7829ce8b18c6fa
sha256:1114fa18ba6be7e5db7728ae747a6bf0aab48fedf3ed9e95aff1f9ce51903698
sha256:181ca82fe4d920b46488d29b507d432dd121d9b2842cf8c720c0939e03e4d6a0
sha256:190b2db337109cb9692cd004aeed4b06fef0c10c700a9ba7939d73e697e3bbcc
sha256:1c4afa44a489abf7c7fe8fa642647a8a2786bf7581486e6a308e1078484784e6
sha256:424ad95540cb66adb9e16d768ccc7010923a9ca09dda1f56e4a08804c2de12e6
sha256:537b4796037a46b64fa39c42f642925704abbaab475e5c72a8fc15c258dc1e85
sha256:69bf797b6b6763938b98139814d8be884693806e0e5a50ac4fbbf11eb45f4f27
sha256:6c7f6a555dee6eebfc5497e84b1cacb4fff035e2101d8421768c0c2db54e8da2
sha256:7dcff293e9f411c63d6f1365795a89ac0058995de0f192ad9fb103ab56533ed3
sha256:86b1c4990ca1c80ad5dc0977fe37ab0f80e0f95e03fe79769b451d97e9f7a8f3
sha256:9d8278ca2542af2ff9cf2d8de439758f6b3bbb84bbe6b7a44edf8080b73d2949
sha256:a5c8632ba0135b465956281008a4f9c2263232d92c020b56e1d839aaa4b74834
sha256:b9fe88bf1364613ec01968480d7cb305d69e3de78bb4a56e3448298ffcc25139
sha256:c0c9c2eaa522dd31901c49a40577a68e3ae02cc75226a248813134046b299099
sha256:c2f4e79836f999ff389b82b8636b807c2baa5b702509b2991e651508519857d5
sha256:cdfb1dfb3ca2f8df9e87e5fa33b91a402a8b4ad0ede164dbdf4c25aded618ed3
sha256:ffd8e3d222cb85f642677642037a0e7886a565babdc0e229cc83147895a8ed2c
sha256:ffecd238fa95b110a1b5f71034b2bd358358758abb52fd098241200d94111979

Where before you wouldn’t bother trying to have your application and database images share layers, with Nix the layer sharing is completely automatic.

The automatic splitting and prioritization has improved image push and fetch times by an order of magnitude. Having multiple images allows Docker to request more than one at a time.

Thank you Target for having sponsored this work with Tweag in NixOS/nixpkgs#47411.

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments.

The Nix community has collected and curated build instructions (expressions) for many thousands of packages in the Nix package collection, Nixpkgs. The collection is a large GitHub repository, which receives over a thousand pull requests each month. Some of these changes can sometimes cause all of the packages to rebuild.

To test changes to Nixpkgs and release updates for Nix and NixOS, we necessarily created our own build infrastructure. This allows us to give better quality guarantees to our users.

The NixOS infrastructure team runs several types of servers: VMs on AWS, bare metal, macOS systems, among others. We build thousands of packages a day, sometimes reaching many tens of thousands per day.

Some of our builds depend on unique features like KVM which are only available by using bare metal servers, and all of them benefit from numerous, powerful cores.

For over a year now, Packet.net (where you can natively deploy NixOS by the way!) has been generously providing bare metal hardware build resources for the NixOS build farm, and together we were curious how the new EPYC from AMD would compare to the hardware we were already using.

For this benchmark we are comparing Packet’s m1.xlarge.x86 against Packet’s first EPYC machine, c2.medium.x86. Hydra already runs a m1.xlarge.x86 build machine, so the comparison will be helpful in deciding if we should replace it with EPYC hardware.

AMD EPYC has the chance to reduce our hardware footprint, reduce our need for our AWS scale-out, and improve our turnaround time for time-sensitive security patches.

System Comparison:

Benchmark Methods

All of these tests were run on a fresh Packet.net server running NixOS 18.03 and building Nixpkgs revision 49a6964a425.

For each test, I ran each build five times on each machine with --check which forces a local rebuild:

checkM() {
  nix-build . -A "$1"
  for i in $(seq 1 5); do
    rm result
    nix-collect-garbage
    /run/current-system/sw/bin/time -ao "./time-$1" \
      nix-build . -A "$1" --check
  done
}

Benchmark Results

Kernel Builds

NixOS builds a generously featureful kernel by default, and the build can take some time. However, the compilation is easy to spread across multiple cores. In what we will further see is a theme, the EPYC beat the Intel CPU with by about five minutes, or about 35% speed-up.

nix-build '<nixpkgs>' -A linuxPackages.kernel

NixOS Tests

The NixOS release process is novel in that package updates happen automatically as an entire cohesive set. We of course test that specific software compiles properly, but we are also able to perform fully automatic integration tests. We test that the operating system can boot, desktop environments work, and even server tests like validating that our MySQL server replication is still working. These tests happen automatically on every release and is a unique benefit to the Nix build system being applied to an operating system.

These tests use QEMU and KVM, and spawn one or more virtual machines running NixOS.

Plasma5 and KDE

The Plasma5 test looks like the following:

1. First launches a NixOS VM configured to run the Plasma5 desktop manager.

2. Uses Optical Character Recognition to wait until it sees the system is ready for Alice to log in and then types in her password:

   $machine->waitForText(qr/Alice Foobar/);
   $machine->sendChars("foobar\n");
   

3. Waits for the Desktop to be showing, which reliably indicates she has finished logging in:

   $machine->waitForWindow("^Desktop ");
   

4. Launches Dolphin, Konsole, and Settings and waits for each window to appear before continuing:

   $machine->execute("su - alice -c 'dolphin &'");
   $machine->waitForWindow(" Dolphin");
   $machine->execute("su - alice -c 'konsole &'");
   $machine->waitForWindow("Konsole");
   $machine->execute("su - alice -c 'systemsettings5 &'");
   $machine->waitForWindow("Settings");
   

5. If all of these work correctly, the VM shuts down and the tests pass.

Better than the kernel tests, we’re pushing a 40% improvement.

nix-build '<nixpkgs/nixos/tests/plasma5.nix>'

MySQL Replication

The MySQL replication test launches a MySQL master and two slaves, and runs some basic replication tests. NixOS allows you to define replication as part of the regular configuration management, so I will start by showing the machine configuration of a slave:

    {
      services.mysql.replication.role = "slave";
      services.mysql.replication.serverId = 2;
      services.mysql.replication.masterHost = "master";
      services.mysql.replication.masterUser = "replicate";
      services.mysql.replication.masterPassword = "secret";
    }

1. This test starts by starting the $master and waiting for MySQL to be healthy:

   $master->start;
   $master->waitForUnit("mysql");
   $master->waitForOpenPort(3306);
   

2. Continues to start $slave1 and $slave2 and wait for them to be up:

   $slave1->start;
   $slave2->start;
   $slave1->waitForUnit("mysql");
   $slave2->waitForUnit("mysql");
   $slave1->waitForOpenPort(3306);
   $slave2->waitForOpenPort(3306);
   

3. It then validates some of the scratch data loaded in to the $master has replicated properly to $slave2:

   $slave2->succeed("
        echo 'use testdb; select * from tests' \
            | mysql -u root -N | grep 4
   ");
   

4. Then shuts down $slave2:

   $slave2->succeed("systemctl stop mysql");
   

5. Writes some data to the $master:

   $master->succeed("
        echo 'insert into testdb.tests values (123, 456);' \
            | mysql -u root -N
   ");
   

6. Starts $slave2, and verifies the queries properly replicated from the $master to the slave:

   $slave2->succeed("systemctl start mysql");
   $slave2->waitForUnit("mysql");
   $slave2->waitForOpenPort(3306);
   $slave2->succeed("
        echo 'select * from testdb.tests where Id = 123;' \
            | mysql -u root -N | grep 456
   ");
   

Due to the multiple VM nature, and increased coordination between the nodes, we saw a 30% increase.

nix-build '<nixpkgs/nixos/tests/mysql-replication.nix>'

BitTorrent

The BitTorrent test follows the same pattern of starting and stopping a NixOS VM, but this test takes it a step further and tests with four VMs which talk to each other. I could do a whole post just on this test, but in short:

• a machine serving as the tracker, named $tracker.
• a client machine, $client1
• another client machine, $client2
• a router which facilitates some of the incredible things this test is actually doing.

I’m going to gloss over the details here, but:

1. The $tracker starts seeding a file:

   $tracker->succeed("opentracker -p 6969 &");
   $tracker->waitForOpenPort(6969);
   my $pid = $tracker->succeed("transmission-cli \
        /tmp/test.torrent -M -w /tmp/data");
   

2. $client1 fetches the file from the tracker:

   $client1->succeed("transmission-cli \
        http://tracker/test.torrent -w /tmp &");
   

3. Kills the seeding process on tracker so now only $client1 is able to serve the file:

   $tracker->succeed("kill -9 $pid");
   

4. $client2 fetches the file from $client1:

   $client2->succeed("transmission-cli \
        http://tracker/test.torrent -M -w /tmp &");
   

If both $client1 and $client2 receive the file intact, the test passes.

This test sees a much lower performance improvement, largely due to the networked coordination across four VMs.

nix-build '<nixpkgs/nixos/tests/bittorrent.nix>'

The remarkable part I left out is Client1 uses UPnP to open a port of the firewall of the router which Client2 uses to read from Client1.

Standard Environment

Our standard build environment, stdenv, is at the deepest part of this tree and almost nothing can be built until after it is completed. stdenv is like build-essential on Ubuntu.

This is an important part of the performance story for us: Nix builds are represented by a tree, where Nix will schedule as many parallel builds as possible as long as its parents are done building. Single core performance is the primary factor impacting how long these builds take. Shaving off even a few minutes means our entire build cluster is able to get to share the work sooner.

For this reason, the stdenv test is the one exception to the methodology. I wanted to test a full build from bootstrap to a working standard build environment. To force this, I changed the very root build causing everything “beneath” it to require a rebuild by applying the following patch:

diff --git a/pkgs/stdenv/linux/default.nix b/pkgs/stdenv/linux/default.nix
index 63b4c8ecc24..1cd27f216f9 100644
--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -37,6 +37,7 @@ let

   commonPreHook =
     ''
+
       export NIX_ENFORCE_PURITY="''${NIX_ENFORCE_PURITY-1}"
       export NIX_ENFORCE_NO_NATIVE="''${NIX_ENFORCE_NO_NATIVE-1}"
       ${if system == "x86_64-linux" then "NIX_LIB64_IN_SELF_RPATH=1" else ""}

The impact on build time here is stunning and makes an enormous difference: almost a full 20 minutes shaved off the bootstrapping time.

nix-build '<nixpkgs>' -A stdenv

Conclusion

This EPYC machine has made a remarkable improvement in our build times and is helping the NixOS community push timely security updates and software updates to users and businesses alike. We look forward to expanding our footprint to keep up with the incredible growth of the Nix project.

Thank you to Packet.net for providing this hardware free of charge for this test through their EPYC Challenge.

and thank you Gustav, Daiderd, Andi, and zimbatm for their help with this article

I heard several times from users in Hong Kong and Singapore that the cache was “slow”, but I didn’t know it was this slow! After working closely with a team of Nix users in Bangalore, I experienced first-hand just how eye-wateringly slow it could be.

The NixOS cache is now being served from all of AWS CloudFront edge locations, significantly reducing latency for users in Asia, Africa, South America, and Oceania.

By expanding the cache’s distribution settings to include all of the edge locations, performance has been substantially improved build time:

Experiments in Tokyo and Hong Kong produced similar results.

NixOS’s cache is stored in AWS S3, and distributed using AWS CloudFront. This combination gives us the excellent durability guarantees of S3 combined with the large geographical distribution of CloudFront.

Until today, the NixOS cache was only served through edge nodes in the United States, Canada, and Europe.

A big thank-you to Amine Chikhaoui, and Eelco Dolstra for their help in researching this change and turning on such a massive improvement.

First, configure Prometheus’s NodeExporter to enable Textfile collection:

services.prometheus.nodeExporter = {
  enable = true;
  enabledCollectors = [
    "textfile"
  ];
  extraFlags = [
    "--collector.textfile.directory=/var/lib/prometheus-node-exporter-text-files"
  ];
};

Second, populate the textfile directory with the current system version on every boot and deployment:

system.activationScripts.node-exporter-system-version = ''
  mkdir -pm 0775 /var/lib/prometheus-node-exporter-text-files
  (
    cd /var/lib/prometheus-node-exporter-text-files
    (
      echo -n "system_version ";
      readlink /nix/var/nix/profiles/system | cut -d- -f2
    ) > system-version.prom.next
    mv system-version.prom.next system-version.prom
  )
'';

Then, configure Grafana to use the system_version as an Annotation with the following query:

changes(system_version[5m])

Repair manual: http://topics-cdn.dell.com/pdf/xps-15-9560-laptop_setup%20guide_en-us.pdf

I have:

Product Name: XPS 15 9560
System BIOS: 1.0.3
ePSA: Build 4304.17 UEFI ROM

Initial Setup

Make a USB recovery disk for windows

You need a 16gb flash drive, and it will use the whole drive. Search Windows for “recovery” and it should recommend “Create a recovery drive” or something.

Make a note on the drive, store it for later. Just in case.

Also write the installer image to a USB disk ask described in the instructions: https://nixos.org/nixos/manual/index.html#sec-uefi-installation

Plug in your USB disk before rebooting to the BIOS.

Reboot to the BIOS

Pressing F2 when the DELL logo appears on the screen (repair manual, page 96) takes you to the BIOS.

Pressing F2 before the DELL logo appears enters you in to a Diagnostic Boot.

Note: You don’t need to press “Fn” with “F2” to enter setup.

You can navigate the BIOS with the arrow keys, or mouse, or touch screen.

Disable Secure Boot

Navigate to Secure boot -> Secure Boot Enable, select Disabled.

In General -> Boot Sequence, you should see in the top right “Windows Boot Manager” and an entry for your USB drive, “UEFI: …”. Select the Windows Boot Manager by tapping on it or with the mouse, then using the arrow icons to right move it to the bottom of the priority list.

When I did this, I also inserted my Windows Recovery USB and made it the highest priority. This way, I can check the disk is good, reboot, unplug, and move on to the NixOS without going back in to the BIOS.

This seemed to only make the change for a single boot. I also always saved these as User Custom Settings or whatever the option is. I’m not sure what that means, but I did it.

Disable RAID mode

System Configuration -> SATA Operation -> Select AHCI.

In RAID mode, Linux doesn’t seem to pick them up. I’m not sure why it is in RAID mode to begin with. If you want to dig in to this, the hardware reported here as Intel 82801 SATA in RAID mode.

Optional: Enable USB Wake Support

Power Management -> USB Wake Support -> Enable USB Wake Support

Interesting Features

• System Configuration -> USB PowerShare
• Security -> Various password settings (including Strong Password and Password Bypass)
• Security -> TPM 2.0 Security
• Intel Software Guard Extensions
• C-States Control?
• Turboboost?
• MEBx Hotkey

According to Security -> OROM Keyboard Access, you can press:

• Ctrl-1 to access Intel RAID config
• Crll-P/F12 to access Intel Management Engine BIOS Extension

Reboot in to the NixOS Installer

I reboot and there it is, loaded, the beautiful (lol) systemd-boot. The text is very small. God help our eyes.

At the boot menu, I selected the default, “NixOS Live CD”. It worked perfectly. I also tried the “NixOS Live CD (with nomodeset)” and it seemed to work just the same. I continued on with “NixOS Live CD”.

Resolution….

I tried pressing e to edit boot options and set the screen to a lower resolution with vga and video, but none of them worked. One point for debugging is I see in the journal:

fb0: EFI VGA frame buffer device

and if I looked at /sys/class/graphics/fb0/modes I only saw one mode:

3840x2160p-87

Windows reports a resolution of 3840x2160, but scaled 250%.

After you boot, you can type setfont latarcyrheb-sun32 and get a slightly larger font. (Found: https://wiki.archlinux.org/index.php/HiDPI#Linux_console)

Wifi

Works out of the box! Yay! Uses an Atheros driver (model QCA6174)

can be setup with:

wpa_supplicant -D nl80211,ext -i wlp2s0 -c <(wpa_passphrase YourWififNetwork "YourWifiPassword")

After connecting, Fn-Alt-F2 to change to TTY2 for the rest of the work. (Fix the font there, too.)

This was extremely helpful: https://github.com/NixOS/nixpkgs/issues/13342#issuecomment-301451545

Disks

If you don’t have devices at /dev/nvme_* you forgot to turn off RAID mode, or you’re on your own…

You’ll want to use gdisk on /dev/nvme0n1, I found this via lsblk, as according to the UEFI instructions in the manual.

I then deleted partitions 2 through 6, leaving only 500M EFI system partition, and created partitions so it looked like:

Note I use 8300 as the code because they’re all encrypted. Calling the swap partition swap, systemd will try to automatically use it.

The cryptsetup luks key partition is 3 MiB so that it can fit the default 2 MiB LUKS header as well as the key.

Then:

# Create an encrypted disk to hold our key, the key to this drive
# is what you'll type in to unlock the rest of your drives... so,
# remember it:
$ cryptsetup luksFormat /dev/nvme0n1p2
$ cryptsetup luksOpen /dev/nvme0n1p2 cryptkey

# Fill our key disk with random data, wihch will be our key:
$ dd if=/dev/urandom of=/dev/mapper/cryptkey bs=1024 count=14000

# Use the encrypted key to create our encrypted swap:
$ cryptsetup luksFormat --key-file=/dev/mapper/cryptkey /dev/nvme0n1p3

# Create an encrypted root with a key you can remember.
$ cryptsetup luksFormat /dev/nvme0n1p4
# Now add the cryptkey as a decryption key to the root partition, this
# way you can only decrypt the cryptkey on startup, and use the
# cryptkey to decrypt the root.
#
# The first human-rememberable key we added is just in case.
$ cryptsetup luksAddKey /dev/nvme0n1p4 /dev/mapper/cryptkey

# Now we open the swap and the root and make some filesystems.
$ cryptsetup luksOpen --key-file=/dev/mapper/cryptkey /dev/nvme0n1p3 cryptswap
$ mkswap /dev/mapper/cryptswap

$ cryptsetup luksOpen --key-file=/dev/mapper/cryptkey /dev/nvme0n1p4 cryptroot
$ mkfs.ext4 /dev/mapper/cryptroot

# and rebuild the boot partition:
$ mkfs.vfat /dev/nvme0n1p1

Then for a not fun bit, matching entries in /dev/disk/by-uuid/ to the partitions we want to mount where. Running ls -l /dev/disk/by-uuid/ shows which devices have which UUIDs. To determine what dm-1 and dm2, I ran ls -la /dev/mapper:

Note I do have a dm-0 for cryptkey, but no UUID but we won’t need it. I substituted the actual hash with A s B s C s D s E s and F s in order to make the mount commands easier.

# Enable swap using the decrypted cryptswap:
$ swapon /dev/disk/by-uuid/EEEEEEEE-EEEE-EEEE-EEEE-EEEEEEEEEEEE

# Mount the decrypted cryptroot to /mnt
$ mount /dev/disk/by-uuid/FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF /mnt

# Setup and mount the boot partition
$ mkdir /mnt/boot
$ mount /dev/disk/by-uuid/AAAA-AAAA /mnt/boot

Initial Configuration

Run nixos-generate-config --root /mnt

hardware-configuration.nix changes

I had to edit the hardware-configuration.nix to setup the luks configuration. I did this with nix-shell -p emacs, deleted the boot.initrd.luks.devices line, and added:

{
  # !!! cryptkey must be done first, and the list seems to be
  # alphabetically sorted, so take care that cryptroot / cryptswap,
  # whatever you name them, come after cryptkey.
  boot.initrd.luks.devices = {
    cryptkey = {
      device = "/dev/disk/by-uuid/BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB";
    };

    cryptroot = {
      device = "/dev/disk/by-uuid/DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD";
      keyFile = "/dev/mapper/cryptkey";
    };

    cryptswap = {
      device = "/dev/disk/by-uuid/CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC";
      keyFile = "/dev/mapper/cryptkey";
    };
  };
}

It should already be correct, but check that:

1. swapDevices refers to /dev/disk/by-uuid/EEEEEEEE-EEEE-EEEE-EEEE-EEEEEEEEEEEE
2. fileSystems."/boot".device refers to /dev/disk/by-uuid/AAAA-AAAA
3. fileSystems."/".device refers to /dev/disk/by-uuid/FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF

configuration.nix changes

The generated configuration.nix seemed correct, too, but I made some changes:

{
  # I set the boot.kernelPackages to linuxPackages_latest just out of
  # habit, since I'm a power-user I'd rather detect problems before
  # other users.
  boot.kernelPackages = pkgs.linuxPackages_latest;

  # I like networkmanager over wpa_supplicant, and with networkmanager
  # you don't need to enable networking.wireless. You _do_ need to
  # enable one of them, or you'll not have wifi when you boot back up.
  networking.networkmanager.enable = true;

  i18n.consoleFont = "latarcyrheb-sun32";

  # See the cpufreq section below
  powerManagement.cpuFreqGovernor = "powersave";

  services.xserver = {
    enable = true;
    # Note I didn't set autorun to be true until I figured out the
    # monitorSection / went through the "X Server Resolution" process
    # below, because the xserver display was much too small. I set it
    # to true after I had sorted out the DPI.
    autorun = true;

    # libinput performs better for me than synaptics:
    libinput.enable = true;

    # Just my personal preference:
    displayManager.lightdm.enable = true;
    windowManager.i3.enable = true;

    monitorSection = ''
      DisplaySize 406 228
    '';
  };
}

I did not set users.mutableUsers to false yet, or create my own user. I usually do this once I get to a GUI so I can set hashedPassword.

X Server Resolution

Note that when testing this, I would:

1. run systemctl start display-manager to start it
2. Press escape to exit the i3 configurator
3. press Alt-Enter to get a terminal
4. run systemctl stop display-manager to stop X
5. type Fn-Alt-F1 to get back to my terminal

xdpyinfo | grep -B2 resolution (installed with nix-shell -p xorg.xdpyinfo) revealed:

screen #0:
 dimensions:  3840x2160 pixels (1016x571 millimeters)
 resolution:  96x96 dots per inch

Successfully raised the resolution:

I took the 1016x571 and divided by 2.5, based on the 250% scaling I saw in the Windows settings, and using https://wiki.archlinux.org/index.php/Xorg#Display_size_and_DPI, came up with:

{
  services.xserver.monitorSection = ''
    DisplaySize 406 228
  '';
}

however when I started a terminal in i3, it was still tiny. Adding terminator to systemPackages seemed to scale correctly. Note: on my other High-DPI computer, I never fixed xterm and it never seemed to matter.

Failed to fix the resolution:

{
  services.xserver.screenSection = ''
    Option "DPI" "240 x 240"
  '';
}

Fixed the resolution, but was super fuzzy:

On startup, running xrandr --output eDP1 --scale 0.4x0.4 (0.4 being 1 divided by 2.5)

Install…

NOTE: If you use nix-shell you must not run nixos-install inside the nix-shell!

I ran nixos-install, rebooted, and it worked like a charm.

Boots fine!

Hardware Check

Battery sensor worked out of the box

Webcam looks up your nose, but worked fine out of the box.

Backlight (intel) works out of the box

$ cat /sys/class/backlight/intel_backlight/max_brightness
187

$ echo "100" | sudo tee /sys/class/backlight/intel_backlight/brightness

$ nix-shell -p xorg.xbacklight --run "xbacklight -set 50"

See: https://wiki.archlinux.org/index.php/Backlight

Audio works out of the box, including headphone switching

Volume control: alsamixer works out of the box, using Master / Headphone / Speaker mixers.

Suspend

systemctl suspend suspects correctly, and pressing the mouse button resumes correctly.

Hibernate

systemctl hibernate great if you have swap.

Trackpad

Using services.xserver.libinput.enable = true; fixes all these problems, and the touch screen continue to work. Right click by clicking in the bottom right corner.

Notes from before:

It works, but it seems to be in absolute positioning mode, and right click seems to not work.

psmouse serio1: synaptics: device claims to have extended capability 0x0c, but I'm not able to read it.
psmouse serio1: synaptics: Unable to initialize device.

Later:

psmouse serio1: Failed to enable mouse on isa0060/serio1

=ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored=

Not an issue: https://askubuntu.com/questions/175793/what-does-the-following-dmesg-output-means

cpupower

cpupower: Setting cpu: 0
cpupower: Error setting new values.

Add: powerManagement.cpuFreqGovernor = "powersave"; to your configuration. See: https://github.com/NixOS/nixpkgs/issues/9611

BIOS Upgrades Without Windows

1. Make sure you’re on AC power. NOT the optional battery pack! It will not boot to USB on the battery pack.
2. nix-shell -p gparted -p unetbootin
3. run sudo gparted a format your USB disk to have one partition, FAT32, with the boot flag. After, right click and Format the partition as fat32.
4. Mount your USB disk (mine was at /dev/sda1) to /mnt
5. Run sudo unetbootin
6. Select FreeDOS 1.0 from the menu
7. Make sure the USB Drive Type is selected, and select the USB drive (again, mine was /dev/sda1) Then press OK
8. Download your BIOS update from Dell, and copy it to /mnt/.
9. Reboot to the BIOS setup
10. under Boot Options change UEFI to Legacy and make sure the boot devices above included at least USB.
11. Prepare to stick around for the next 4 steps! Don’t look away! FreeDOS’s default option is install to the hard disk! It may be fine, but I’ve never tried it.
12. Save and quit with the USB device plugged in.
13. Select the only boot option
14. Then select the “safe mode” option that says it won’t load drivers.
15. Type C:<enter> then dir<enter> and type XPS<tab> to auto-select the XPS BIOS update and press Enter.
16. The update will prompt for y a few times. Once it is done, enter the BIOS again.
17. Verify the update took place
18. Disable the legacy boot, switching back to UEFI.

Things to look in to…

Touch screen

• Touch screen works out of the box (WHAT!)
• doesn’t scroll?
• any touch is a click

In /proc/bus/input/devices the touchscreen is labeled:

I: Bus=0003 Vendor=04f3 Product=24a0 Version=0110
N: Name="ELAN Touchscreen"
P: Phys=usb-0000:00:14.0-9/input0
S: Sysfs=/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.0/0003:04F3:24A0.0001/input/input11
U: Uniq=
H: Handlers=event6 mouse0
B: PROP=2
B: EV=2
B: KEY=400 0 0 0 0 0
B: ABS=3273800000000003 (may be wrong number of 0's?)

Running cat /dev/input/event6 revealed lots of output when I touched the screen.

Under xinput it is id=12. xinput list-props 12 revealed more information.

See:

• https://wiki.archlinux.org/index.php/Touchscreen
• https://wiki.archlinux.org/index.php/Libinput
• https://askubuntu.com/questions/863563/libinput-and-touchscreen-support

VGA Switcheroo

Unable to enable Bumblebee, because I’m using the latest linux kernel and the nvidia driver doesn’t compile with it yet.

VGA Switcheroo: detected Optimus DSM method \_SB_.PCI0.PEG0.PEGP handle
nouveau: detected PR support, will not use DSM
nouveau: 0000:01:00.0: enabling device (0006 -> 0007)
nouveau: 0000:01:00.0: unknown chipset (137000a1)
nouveau: probe of 0000:01:00.0 failed with error -12

Wifi

athk10k_pci 0000:02:00.0: Direct firmware load for athk10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
athk10k_pci 0000:02:00.0: Direct firmware load for athk10k/cal-pci-0000:02:00.0.bin failed with error -2
athk10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA6174/hw3.0/firmware-5.bin failed with error -2
athk10k_pci 0000:02:00.0: could not fetch firmware file 'ath10k/QCA6174/hw3.0/firmware-5.bin': -2

Warnings pre crypt-setup

ACPI Error: [\_SB_.PCIO.XHC_.RHUB.HS11] Namespace lookup failure, AE_NOT_FOUND (20160831/dswload-210)
ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20160831/psobject-227)
ACPI Exception: AE_NOT_FOUND, (SSDT:xh_rvp11) while loading table (20160831/tbxfload-228)
ACPI Error: 1 table load failures, 10 successful (20160831/tbxfload-246)
DMAR: DRHD: handling fault status reg 2
DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index 2010 [fault reason 34] Present field in the IRTE entry is clear

mce hardware error

mce_notify_irq: 1 callbacks supressed
mce: [Hardware Error]: machine check events logged

Some reports of these here: http://en.community.dell.com/techcenter/os-applications/f/4613/t/19997490

Using mcelog (nix-shell -p mcelog) it shows some output, but requires an update. Our mcelog is out of date. I updated it in unstable, and created an mcelog service. Nothing too scary in the log, not sure what it is.

See: [https://github.com/andikleen/mcelog/blob/master/mcelog.service](https://github.com/andikleen/mcelog/blob/master/mcelog.service)

int3403 thermal: probe of INT3403:06 failed with error -22

Other Notes

• In the console, closing the lid causes it to go to sleep in some fashion, but opening the lid again doesn’t get the screen back.
• In X with no extra configuration, closing the lid and opening it sleeps and then restores the display correctly.

Non-Problems

What are “Dell DMI hotkeys”?

output after entering the crypt password

However, I believe NixOS provides the most succinct and reproducible instructions yet, without even having to muck with defaults write.

Incorporate the following in to your configuration.nix, nixos-rebuild switch, and you’re done:

    {
      networking.firewall.allowedTCPPorts = [
        548 # netatalk
      ];

      services = {
        netatalk = {
          enable = true;

          volumes = {
            "grahamc-time-machine" = {
              "time machine" = "yes";
              path = "/home/grahamc/time-machine";
              "valid users" = "grahamc";
            };
          };
        };

        avahi = {
          enable = true;
          nssmdns = true;

          publish = {
            enable = true;
            userServices = true;
          };
        };
      };
    }

Tested on NixOS 17.03.