NixOS: A Field Report

Why?

NixOS is a relatively new Linux distribution based on the Nix package manager. A Nix package is hashed based on its content, dependencies, compiler , etc. and stored immutably in the Nix store. The Nix package manager uses this Nix store, a patched dynamic linker, and a buttload of symlinks to isolate installs. You can have multiple versions of applications and libraries living on the same machine without conflict.

The Nix package manager also maintains a history of installations and—since the Nix store is full of immutable installs of applications and libraries—Nix provides a way to rollback installs to a previously working configuration. NixOS extends this concept all the way to the Linux kernel. When you change the system configuration for your machine you get a new GRUB entry for that entire profile of your computer: kernel, applications, libraries, and even your configuration files. If something goes badly you just reboot and select the GRUB entry for your previous configuration. Check out section 1.4 of the NixOS manual for some cool details http://nixos.org/nixos/manual/#sec-upgrading.

But wait there's more! These installations are atomic, so it either completes successfully or not at all. So basically what we get here is something like an ACID MVCC for the software on your computer.

There are definitely practical benefits to all this. There have been times that I've done some riskier upgrades. Moving to a new Ubuntu release is one of those times. There's always a bit of a question as to what may or may not be broken. I usually back up my files and do a fresh install. I've also been bitten a couple of times by a bad video driver upgrade that was annoying to recover from. That said, I don't know that I've really needed an ACID MVCC for software all that often.

Aside from riskier installs you can also setup a Nix configuration for each project you work on. You can chroot into this Nix environment. The Nix configuration can be version controlled, so that when you checkout an old branch you can use the environment that matches that version of your project. You can also take this Nix configuration and create a VM from it, or send it off to an EC2 node. Admittedly you can do all of this stuff just using the Nix package manager on Linux or OSX without needing NixOS. (Here is how http://zef.me/5966/setting-up-development-environments-with-nix.)

So, really...Why?

So all this stuff is great, but maybe not worth switching to NixOS for your day-to-day desktop machine. I have moved for a couple of reasons. Lately I've been getting disillusioned with the direction Ubuntu is headed. I barely made it through the last LTS upgrade. I prefer to use Gnome and XMonad, but Ubuntu comes with Unity these days and it's hard to get a plain Gnome install (though I made it work). The flip side is that Ubuntu is a really smooth install where most everything just works out of the box, but that's not really enough to keep me there. I'm basically forced to consider some kind of a major change whether it is a change in distribution or the tools I use or something.

I also like to try to change things up every once in a while. I've been on Ubuntu for close to 10 years and Gnome+XMonad for maybe 3 years. I need to shake things up a bit, learn some new stuff, and feel like a hopeless newb. Boy did I ever accomplish that!

Attempt 1: No Gnome & Miserable Failure with Wireless

I downloaded and burned the NixOS 13.10 minimal CD. It booted up fine and gives you hints to walk you through the process. The CD boots with a copy of the manual on one of the Linux consoles so you can jump back and forth to reference the manual as you're installing. The process is pretty simple, too. You format a partition, mount it, run a config generation program, and customize the generated config. Once the config is set you run an install command and boot into your new NixOS installation.

However, Gnome doesn't install. Based on a reading of the docs you might be led to think that it does, but after 13.10 was released this commit was made to the Nix packages: https://github.com/NixOS/nixpkgs/commit/a734f32fa1efce21a008e391cfb10695c4d738cd, so it's not even an option in more recent versions of NixOS.

Coincidentally XMonad doesn't install either. Well it seems like it does, and if you use it without any custom config then it's probably fine, but to use a custom config you need a Haskell compiler and the XMonad packages. For some reason when the XMonad packages install they don't install the dependencies that they need (which seems broken given how Nix is supposed to work). I had to iteratively run `ghc-pkg check` and install a bunch of packages manually. Perhaps I was missing something. Also, not to give away the ending, but this was with 13.10, so it may be a different story with the latest unstable NixOS.

Finally, I had wireless issues. The installation CD booted and I was able to get an install, but once I tried to find some combination of CLI and/or GUI software to configure my wireless adapter I wedged something, and decided to just reboot and reinstall, but the installation CD was hanging when starting the WPA supplicant. It seemed maybe there was some state stored in the wireless card that was causing the WPA supplicant to hang, but I couldn't boot NixOS in order to configure the card. After many hours fiddling with this to try to get the wireless working and being very frustrated I very nearly gave up on NixOS. As a last ditch effort I decided to try using the unstable version of NixOS.

Attempt 2: Using Unstable NixOS

Out of the gate Unstable NixOS was more stable than the 13.10 release. I was able to actually boot the thing! I made it through the installation process. I had decided that I was going to use Xfce+i3 instead of Gnome+XMonad. Changing my desktop and window manager at the same time as changing my distribution was going to make this whole process more "interesting."

Configuring NixOS is relatively simple once you know what the configuration options are, but finding them isn't as easy as it could probably be. The NixOS manual has a section of configuration options at the end. That was an important reference. I don't know if it documents every possible option, so sometimes I would take a look at the Nix package source. There are also configuration options that affect some packages that aren't entirely obvious unless you look at the package source. I think in the future it would be nice (in addition to the way the documentation is currently organized) to have documentation that can be browsed per package, so you could see for a particular package what options affect it.

Having a working distribution is since, but there were still some annoyances. Searching for packages is also a bit unsatisfying. The recommended way to search seems to be `nix-env -qaP '*' | grep NEEDLE` which works, but it's not the most performant, and you also can only search by package name; you can't search by description or configuration options or anything else. Even when you've found the package you want more detail on, it is not obvious how to find the package. You can browse the nixpkgs source, but it is organized by categories. The best way seems to be doing a `find / -name NEEDLE` or something. None of this is really a deal breaker (for me), but just lessons learned on how to find information.

Finally, on the subject of finding information the nix-dev mailing list is a good resource. I ended up searching it with google "site:http://lists.science.uu.nl/pipermail/nix-dev/ NEEDLE". I guess it is also possible to search the Gmane archive http://news.gmane.org/gmane.linux.distributions.nixos.

Adaptations

There were definitely tools that I'm used to using that were available and worked great on NixOS, among them were Conkeror, Emacs, OpenJDK, Ruby, Tmux, and Skype.

There were replacements I had to make. Obviously Gnome+XMonad became Xfce+i3, and this had a cascading effect.

• I had a raise-or-run script that used wmctrl to activate a program's window, but wmctrl doesn't work with i3, so I had to replace that script with one that uses i3msg.
• I spent a bit of time tuning my i3 config to have bindings similar to the ones I was using with XMonad.
• Previously I was using gnome-panel for workspaces, wireless control, sound volume control, date/time, etc.; that got replaced by i3bar and i3status.
• I was using swarp to shuffle the mouse cursor out of the way in Gnome+XMonad, and that package wasn't available on NixOS, so I replaced that functionality with xdotool.
• I tried to get NetworkManager up and running, and there appear to be configuration options to get it set it up on NixOS, but I ran into issues. I ended up just installing the wpa_gui tool to manage my wireless connections.

I also had a cache of scripts that I used for various things. Some of them were for building programs from source, but I think I can drop those and just use Nix to accomplish the same thing. Others I had to replace '#!/bin/bash' with '#!/usr/bin/env bash' since NixOS doesn't have things at fixed, standard locations like other Linux distributions.

There are some other minor issues to work out. For one my sound card seems to be detected just fine, and Skype will use it, but Xfce doesn't seem to be playing any sounds. I just get a system beep for "sound" events. I think this is an issue with the configuration of Xfce and its sound theme or something. Not really a major issue for me, but I'd like to figure it out eventually.

I would also like to get Dropbox installed. It actually worked fine on 13.10, but for some reason it isn't working so well on unstable. I haven't had a lot of time to look into it, and again not a high priority. I may take the opportunity to look into OwnCloud instead.

Conclusion

I went into this expecting it to be rough at times, and expecting to make some changes to the tools I was used to using. That was no surprise. I was a bit disappointed in how the latest official release of NixOS turned out. The flip side is that the Unstable release has worked great so far, and I'm a little more comfortable following the bleeding edge of NixOS given that (theoretically) I should be able to rollback any changes that bork my system. I'm hoping that turns out to be true.

If you're willing to learn something new and possibly snag your sweater on some rough edges, then NixOS could be a fun experience for you. :) If you use some of the same tools as me, then hopefully what I've discovered can help you move to NixOS. If you want something a little more polished, then you may want to wait on NixOS.

References

My configuration.nix
https://gist.github.com/pjstadig/8183688
Dotfile changes moving from Ubuntu+Gnome+XMonad to NixOS+Xfce+i3
https://github.com/pjstadig/new.dotfiles/compare/455374...master

A Proof of the Braininess of "The Simpsons" and "Futurama"

Occasionally the mathematics does wind more deeply into the story, most notably in the 2010 “Futurama” episode “The Prisoner of Benda.” The plot turns on a device called the Mind-Switcher, which performs just the function its name suggests; over the course of the episode, as the apparatus is used with greater and greater abandon, the minds of the characters shuttle from body to body like singles switching bedrooms in a French farce. By the end, not a single consciousness remains in its proper skull. What’s worse, the characters can’t just retrace their steps to reunite each mind with its original body; the Mind-Switcher, having operated on a pair of minds, isn’t allowed to switch the same two minds again.

Ken Keeler, who wrote the episode, realized that in order to get everything sorted out it might be necessary to introduce new characters, whose bodies could be used as waystations through which the minds could find their way home. An ordinary writer would have been content simply to find a way out of the episode. But Keeler became obsessed with the problem in its full generality, finally composing a proof that, no matter how wild the original fiesta of Mind-Switching, the damage can always be repaired once two new people are added to the system. This question may sound abstruse, but the part of math it belongs to — “combinatorial group theory” — is one of the hottest things going at the moment, with major advances popping up everywhere from Paris to Los Angeles. Keeler’s theorem isn’t one of those big advances, but it’s a real theorem, certainly the deepest piece of mathematics ever featured in a prime-time sitcom.

-- Jordan Ellenberg, "Mathematics and Homer Simpson" (http://www.washingtonpost.com/opinions/2013/12/20/ef1bfaa6-5b9a-11e3-bf7e-f567ee61ae21_print.html)

The Notion of "Real-Time"

"I think the current evolution of technical language around web developers has made real-time mean: 'consume information as soon as it is available' and not '_react to the information in a timely manner or this car will crash_'."

-- Alvaro Videla, "Tell me more about your real-time systems" (http://videlalvaro.github.io/2013/12/tell-me-more-about-realtime.html)

Remote != Distributed

In an age when companies seem to be shutting down their remote work programs, I'm here to tell you that a distributed team can work, and when it works well it is a great experience. I have worked on a distributed team for the past three years. Prior to that I worked for four years in an office that had one full-time remote employee and allowed occasional work from home for others.

Working from home and working remotely are not the same thing as working on a distributed team. If there is a center of gravity at an office and a few remote employees orbit that office, then there is a different experience for some people than for others, and there is a different way of working for some people than for others. There is a natural imbalance that an inanimate system would want to equalize, but in a human organization people pull in opposite directions.

A distributed team is a team where no one is physically co-located. There is no center of gravity at an office. There is no imbalance, because everyone has the same experience, and everyone has the same way of working.

When there is an office/remote split, there is a different experience for some than for others. The office people have shared experiences like water cooler chat and birthday celebrations, and the remote people are left out. The office people gather in a conference room for a meeting and dial in the remote people over a crappy speakerphone. The experiences are different which affects morale and cohesion.

On a distributed team everyone has the same experience. You find new ways to create shared experiences that are different from office experiences and unique to distributed teams. You use tools that allow meetings to be high quality experiences for everyone. These shared experiences support morale and cohesion.

An office/remote split means different ways of working for different people. In-office employees may depend on ad hoc hallway conversations, but remote employees are not privy to those. In-office employees have an "out of sight, out of mind" mentality and think that remote employees aren't productive because they aren't communicating with the remote employees. Remote employees feel disconnected and lack direction for the same reasons.

On a distributed team everyone communicates through the same channels, and everyone can see the activity about who is doing what. Since there aren't two different communication channels there isn't an "out of sight, out of mind" problem.

When there are different experiences, different ways of working, different communication channels at the same company, then two different cultures develop, and that is detrimental. A permanent office/remote split is doomed to failure from the beginning, so it's no wonder that companies are killing their remote work programs.

What does it look like to do a distributed team right? I can tell you what has worked for me and my company. I do not expect that our experiences are universal, but I will try to draw some general principles.

Distributed teams that work well:

• Use tools to fix work into a tangible medium of expression (to borrow from copyright law). Office collaboration via ad hoc hallway conversations and whiteboards doesn't work for distributed teams. You have to produce Google Docs, tickets in Jira, etc. These are things to which everyone can have access. Frankly, distributed or not, this is a better way of working than informal ideas banging around in people's heads.
• Communicate through many-to-many channels. Of course there is still a need for one-to-one communication, but using many-to-many communication channels means that everyone can feel a part of the company. Just like others can walk by and join in on hallway conversations, people can observe and contribute to Skype calls and chat rooms. It is also a good idea to have logs of these chats that people can catch up on (see above).
• Adopt processes that encourage collaboration and productivity. Working on a distributed team may not be for everyone; to a certain extent it does take a self-motivated individual. However, pair programming and daily standups create an environment that encourages collaboration and accountability. We also tend to hire people mostly in North and South America so that the timezones line up better for collaboration. We also meet face-to-face 3-4 times a year, which helps to develop the personal relationships that are necessary to working well together.

Realize that the idea of "remote" working does not work when combined with an office culture and a center of gravity in one location. Working from home or remotely may work temporarily, but eventually gravity will pull you back towards the office. If you try to maintain this unnatural office/remote balance you will enforce different experiences and methods of work and create two different cultures in your company. This cultural divide will lead to jealousy, resentment and other morale and cohesion problems.

This does not mean that it is impossible to function unless everyone reports to the same office. It just means that different tools and processes must be developed to facilitate a distributed team. You must use tools and processes that encourage tangible expression of ideas, many-to-many communication, and collaboration and productivity.

Remote working is isolating and doomed to failure, but a distributed team done right is a joy.