Monday, May 9, 2016

The March of Technology

"Our inventions are wont to be pretty toys, which distract our attention from serious things. They are but improved means to an unimproved end, an end which it was already but too easy to arrive at; as railroads lead to Boston or New York. We are in great haste to construct a magnetic telegraph from Maine to Texas; but Maine and Texas, it may be, have nothing important to communicate. Either is in such a predicament as the man who was earnest to be introduced to a distinguished deaf woman, but when he was presented, and one end of her ear trumpet was put into his hand, had nothing to say. As if the main object were to talk fast and not to talk sensibly. We are eager to tunnel under the Atlantic and bring the Old World some weeks nearer to the New; but perchance the first news that will leak through into the broad, flapping American ear will be that the Princess Adelaide has the whooping cough. After all, the man whose horse trots a mile in a minute does not carry the most important messages; he is not an evangelist, nor does he come round eating locusts and wild honey. I doubt if Flying Childers ever carried a peck of corn to mill."
Thoreau, Henry David. Walden, and on the Duty of Civil Disobedience. Project Gutenberg. Web. 09 May 2016.

Or in the words of a more modern philosopher and poet:

Saturday, March 5, 2016

Making Fake Things

Software is fake. There are bits inside a computer represented by a magnetic or electrical charge or mechanical potential or some such thing. But software is not an electrical charge. Electrical charges can represent ones and zeroes and a series of ones and zeroes like "10111101" can represent the JVM opcode "anewarray" or the fraction one-half ("½") in the ISO-8859-1 character encoding or the number -67 in twos-complement. Software is not electrical charges, it is a particular interpretation imposed on electrical charges. An interpretation does not weigh anything. It has no color, taste, temperature, volume, mass, or any physical features. It is fake, but fake things can be useful.

Fake things can represent real things (or other fake things). For example, you can represent a couch with a 3D model in a computer. You can represent cities and towns and roads with fake things. You can also represent fake things with other fake things. JVM opcodes, characters, and numbers are all fake things represented by "10111101", which is fake. Fake things are useful because they can represent real and fake things in a way that can be cheaply manipulated and transported instantly across the world. Fake things also have challenges.

Software is a little unique even among fake things because in making software we are often making something that has never existed before. When someone creates a stove there are hundreds of thousands of others stoves in existence to draw upon. There are wood stoves, electric stoves, and gas stoves. But when someone created a text editor, they created something that had never existed before. Here is how Richard Gabriel describes it:

"But, consider the first people to design and build a text editor. Before that, there was never a text editor. Changes to a manuscript were always made by retyping or retypesetting. How would people want to make textual changes? How would people want to navigate? Searching? - no one ever heard of that before. Systematic changes? Huh? By the way, there were no display terminals, so how do you even look at the manuscript?" --

Web applications, virtual currencies, automated theorem provers, and many other software applications had never existed before or were so different in nature from their physical counterparts that they were a unique thing. Making fake things is hard enough, but making things that have never existed before is that much harder. That's not the end of it, though.

Fake things have no real world to help co-design them. Stoves have a real world to help co-design them. There are accessories that are used with stoves that help co-design them. Real things like pots and pans. Stoves have to fit through doorways, nestle between kitchen cabinets, and match the colors on the walls. Text editors have accessories like keyboards and mice that were invented to give real people made of meat a way of manipulating a conceptual world by proxy. Perhaps a mouse has to be compatible with a human hand, but a text editor has to be compatible with the mental model of a text editor that exists in a human mind, a model which no one had ever thought of before. Ultimately making software is a process of collaborating with other humans to dream up some mental model, and then making a fake thing out of software that other humans can use to manipulate that model (assuming they properly understand the mental model).

Which reminds me, collaboration is also a fake thing. Collaboration is about using real things, like vibrating air, to push around fake things, like words. It is about using real things, like markers and whiteboards, to manipulate fake things, like ideas. All of these real things can be replaced by fake things, like video conferencing software and text editors. And fake things like words and ideas can be replaced by other fake things, and all of these fake things can be instantly transported, copied, and manipulated by real people in real (and very distant) places. Collaboration is not a real thing, it is a fake thing produced through the interaction of real people thinking creatively.

And making software is a creative act. Writing software is writing instructions to make a computer do something. You must choose the instructions, determine their order, name things. You develop your own style. Writing software is writing words that have effect. Writing software is as close as you can get to God with words speaking reality into existence, the ultimate creative act. But writing software is not just for telling computers what to do. It is also collaboration with other humans. They must read, understand, modify, and extend what you write. They must understand your vision. You must collaborate with them through your source code.

So, here we are. We have discovered that software is a fake thing, that it is often an entirely new thing, that it is a pure product of the mind, that it is born of collaboration, and it is creative expression. Now what? We must systematically question the constraints we place on ourselves, because those constraints are often meant for real things and our things are fake. Here are a few examples:

A top-down management hierarchy is for making real things, not fake things. Top-down, command-and-control hierarchies are about control and efficiency. Control and efficiency are important for real things, because real things have locality, cost, and scarcity. Software has none of these things. Control and efficiency are important when you are manufacturing the same thing over and over. Software is often exploratory. Software is valuable not because we repetitively make lots of little copies of the same thing, but because we dream up some new way of doing things that has never been done before. Control and efficiency are important when you have a predictable process. A creative process is not predictable. You may think for hours about a problem, sleep on it, and then have the answer pop into your head the instant you wake up. We need to think differently, not just about what we make, but how we make it.

Offices are about locality. An office puts materials, means of production, and managers in the same physical location. Yet with software there is no material and the means of production are mental. There is no reason to be concerned about locality. Ostensibly having a bunch of people in the same office enables them to collaborate, but collaboration is a fake thing. Collaboration does not exist in San Francisco or Saint Louis. It does not weigh 1kg. It is not blue. Having an office for collaboration is a rationalization that projects the past onto the future. Is collaboration different using video conferencing and Google Docs than it is using tables and chairs in an office? Yes, because fake things are different than real things. I do not recommend mixing fake things like video conferencing with real things like offices. It may take getting used to, but embracing the fakeness of collaboration has advantages like hiring people where they want to live instead of trying to convince them to live where you live. It also means having permanent, searchable, modifiable artifacts that can be shared instantly across the world, instead of a whiteboard in a room.

Software can process data, but software is also data. This creates leverage. You can flip a bit, and that bit can flip ten others, and those ten another one hundred, etc. Compilers, build tools, continuous integration, and automated tests are all software doing things to software. "The cloud" has created a lot of leverage because it took something that was real (a machine) and made it fake (a "cloud instance"), and once it is fake it can be manipulated by software. The higher you can climb the mountain of abstraction the more powerful you will become. Before selling to Facebook, WhatsApp had ~450 million active users and ~55 employees. Yahoo has ~12,500 employees. I don't know how many active users they have, but let's just pretend it is ~450 million. Don't be Yahoo.

These are just examples, and you can agree or disagree. My point is, we as an industry can achieve market success and realize our visions much more powerfully, but we must understand the nature of the software we are creating (it is fake), and the newness of what we are doing every day, and its collaborative nature, and the tools that we can take advantage of, and we must have the courage to give up on arbitrary constraints that are optimized for making real things. We must pursue leverage, because leverage will enable us to do amazing things.

Friday, May 30, 2014

The following is a quote from an astronaut speaking about a spacecraft

"There's so much more elbow room in there compared to the Soyuz," he said. "Instead of just bringing two of your buddies, you can bring six ... It's got modern electronics, it's got modern materials in the heat shield. So technologically, it's a giant leap beyond the Soyuz."

SpaceX reveals new-look passenger spacecraft

One could think of a number of snarky comments that could be made (especially given that Elon Musk is also the CEO of Tesla), but I'm kind of amazed that this is how human beings are speaking about spacecraft these days.

Perhaps it should all be taken with a grain of salt since this quote was spoken at a publicity event by someone who works for SpaceX. long until I can take a vacation to Mars?

Saturday, May 10, 2014

This, But For Computer Programming

If you meet a philosopher on a train and ask him his profession, he is likely to lie. It is not that philosophers are especially prone to lying, but rather that philosophy is a peculiar profession. To tell your fellow passenger that you are a philosopher opens up an awkward line of questioning. ... If you take the plunge, however, and accept the label of philosopher, you must be prepared for the disappointment when your listener hears that you don't live in a hut on a mountaintop, haven't uncovered the secret of life, and cannot explain why the world exists. If you are foolish enough to go further and attempt to describe your lifelong attempt to reconcile the epistemology of mathematics with its ontology, be prepared to encounter a look in which boredom and horror are blended equally. Best, therefore, to say simply that you are an architect, and leave it at that.

A World Without Time, Palle Yourgrau, page 164.

Some possible substitutions:
PhilosopherComputer Programmer
live in a hut on a mountaintoplive in your parents basement
uncovered the secret of lifeknow why my computer is running so slowly
explain why the world existsinstall the driver for my printer
reconcile the epistemology of mathematics with its ontologyfigure out the best cache invalidation strategy to provide a balance of performance and freshness

As a corollary, I wonder if there is a t-shirt for philosophers that is congruent with the "No, I will not fix your computer" t-shirt for computer professionals, something like "No, I will not fix your worldview."

Tuesday, January 14, 2014

The Solution for Silicon Valley?

"People—especially in the financial community—seem to assume that every industry works kind of like a web startup, and that all you need is two hot guys and $25,000 and you're a millionaire in six months. Heavy semi doesn't work like that. Heavy semi is like steel mills and railroads. By the time you can get a serious semi company self sustaining you're looking at a couple hundred million dollars of investment." — Ivan Godard of Out-of-the-Box Computing (

It's a longer conversation to have, but I have opinions about Silicon Valley (that I share with others). However, it is nice to see people trying to create a real business and solve real problems instead of creating the next Instagram clone. The thing is that it takes hard work, many years, and the perspective to step back and work towards fundamental—not incremental—change.

Best of luck to the OOTBC guys! True, a new CPU architecture isn't exactly solving problems of justice and social inequality, but at least it isn't a get rich quick scheme. Finding ways to fill in computation at neglected price points in the market is a stepping stone to solving other problems. Hopefully those problems aren't how to gamify haircuts or something. *sigh*

Let's have some get-rich-slow startups that are truly innovating to solve difficult, fundamental problems.

Wednesday, January 8, 2014

The Buck Stops Here

"Rufer explains the thinking behind the process: 'I was signing checks one day and I recalled the saying, "The buck stops here." I thought to myself, that isn't true. In front of me was a purchase order, a note that said the stuff had been shipped, we had received it, and that the price on the invoice matched the puchase order. A check had been prepared. Now, do I have the choice not to sign the check? Nope. So the question isn't where the buck stops, but where it starts—and it starts with the person who needs the equipment. I shouldn't have to review the purchase order, and the individual shouldn't have to get a manager's approval.'"

"First Let's Fire All the Managers," Gary Hamel (

Monday, December 30, 2013

NixOS: A Field Report


NixOS is a relatively new Linux distribution based on the Nix package manager. A Nix package is hashed based on its content, dependencies, compiler , etc. and stored immutably in the Nix store. The Nix package manager uses this Nix store, a patched dynamic linker, and a buttload of symlinks to isolate installs. You can have multiple versions of applications and libraries living on the same machine without conflict.

The Nix package manager also maintains a history of installations and—since the Nix store is full of immutable installs of applications and libraries—Nix provides a way to rollback installs to a previously working configuration. NixOS extends this concept all the way to the Linux kernel. When you change the system configuration for your machine you get a new GRUB entry for that entire profile of your computer: kernel, applications, libraries, and even your configuration files. If something goes badly you just reboot and select the GRUB entry for your previous configuration. Check out section 1.4 of the NixOS manual for some cool details

But wait there's more! These installations are atomic, so it either completes successfully or not at all. So basically what we get here is something like an ACID MVCC for the software on your computer.

There are definitely practical benefits to all this. There have been times that I've done some riskier upgrades. Moving to a new Ubuntu release is one of those times. There's always a bit of a question as to what may or may not be broken. I usually back up my files and do a fresh install. I've also been bitten a couple of times by a bad video driver upgrade that was annoying to recover from. That said, I don't know that I've really needed an ACID MVCC for software all that often.

Aside from riskier installs you can also setup a Nix configuration for each project you work on. You can chroot into this Nix environment. The Nix configuration can be version controlled, so that when you checkout an old branch you can use the environment that matches that version of your project. You can also take this Nix configuration and create a VM from it, or send it off to an EC2 node. Admittedly you can do all of this stuff just using the Nix package manager on Linux or OSX without needing NixOS. (Here is how

So, really...Why?

So all this stuff is great, but maybe not worth switching to NixOS for your day-to-day desktop machine. I have moved for a couple of reasons. Lately I've been getting disillusioned with the direction Ubuntu is headed. I barely made it through the last LTS upgrade. I prefer to use Gnome and XMonad, but Ubuntu comes with Unity these days and it's hard to get a plain Gnome install (though I made it work). The flip side is that Ubuntu is a really smooth install where most everything just works out of the box, but that's not really enough to keep me there. I'm basically forced to consider some kind of a major change whether it is a change in distribution or the tools I use or something.

I also like to try to change things up every once in a while. I've been on Ubuntu for close to 10 years and Gnome+XMonad for maybe 3 years. I need to shake things up a bit, learn some new stuff, and feel like a hopeless newb. Boy did I ever accomplish that!

Attempt 1: No Gnome & Miserable Failure with Wireless

I downloaded and burned the NixOS 13.10 minimal CD. It booted up fine and gives you hints to walk you through the process. The CD boots with a copy of the manual on one of the Linux consoles so you can jump back and forth to reference the manual as you're installing. The process is pretty simple, too. You format a partition, mount it, run a config generation program, and customize the generated config. Once the config is set you run an install command and boot into your new NixOS installation.

However, Gnome doesn't install. Based on a reading of the docs you might be led to think that it does, but after 13.10 was released this commit was made to the Nix packages:, so it's not even an option in more recent versions of NixOS.

Coincidentally XMonad doesn't install either. Well it seems like it does, and if you use it without any custom config then it's probably fine, but to use a custom config you need a Haskell compiler and the XMonad packages. For some reason when the XMonad packages install they don't install the dependencies that they need (which seems broken given how Nix is supposed to work). I had to iteratively run `ghc-pkg check` and install a bunch of packages manually. Perhaps I was missing something. Also, not to give away the ending, but this was with 13.10, so it may be a different story with the latest unstable NixOS.

Finally, I had wireless issues. The installation CD booted and I was able to get an install, but once I tried to find some combination of CLI and/or GUI software to configure my wireless adapter I wedged something, and decided to just reboot and reinstall, but the installation CD was hanging when starting the WPA supplicant. It seemed maybe there was some state stored in the wireless card that was causing the WPA supplicant to hang, but I couldn't boot NixOS in order to configure the card. After many hours fiddling with this to try to get the wireless working and being very frustrated I very nearly gave up on NixOS. As a last ditch effort I decided to try using the unstable version of NixOS.

Attempt 2: Using Unstable NixOS

Out of the gate Unstable NixOS was more stable than the 13.10 release. I was able to actually boot the thing! I made it through the installation process. I had decided that I was going to use Xfce+i3 instead of Gnome+XMonad. Changing my desktop and window manager at the same time as changing my distribution was going to make this whole process more "interesting."

Configuring NixOS is relatively simple once you know what the configuration options are, but finding them isn't as easy as it could probably be. The NixOS manual has a section of configuration options at the end. That was an important reference. I don't know if it documents every possible option, so sometimes I would take a look at the Nix package source. There are also configuration options that affect some packages that aren't entirely obvious unless you look at the package source. I think in the future it would be nice (in addition to the way the documentation is currently organized) to have documentation that can be browsed per package, so you could see for a particular package what options affect it.

Having a working distribution is since, but there were still some annoyances. Searching for packages is also a bit unsatisfying. The recommended way to search seems to be `nix-env -qaP '*' | grep NEEDLE` which works, but it's not the most performant, and you also can only search by package name; you can't search by description or configuration options or anything else. Even when you've found the package you want more detail on, it is not obvious how to find the package. You can browse the nixpkgs source, but it is organized by categories. The best way seems to be doing a `find / -name NEEDLE` or something. None of this is really a deal breaker (for me), but just lessons learned on how to find information.

Finally, on the subject of finding information the nix-dev mailing list is a good resource. I ended up searching it with google "site: NEEDLE". I guess it is also possible to search the Gmane archive


There were definitely tools that I'm used to using that were available and worked great on NixOS, among them were Conkeror, Emacs, OpenJDK, Ruby, Tmux, and Skype.

There were replacements I had to make. Obviously Gnome+XMonad became Xfce+i3, and this had a cascading effect.
  • I had a raise-or-run script that used wmctrl to activate a program's window, but wmctrl doesn't work with i3, so I had to replace that script with one that uses i3msg.
  • I spent a bit of time tuning my i3 config to have bindings similar to the ones I was using with XMonad.
  • Previously I was using gnome-panel for workspaces, wireless control, sound volume control, date/time, etc.; that got replaced by i3bar and i3status.
  • I was using swarp to shuffle the mouse cursor out of the way in Gnome+XMonad, and that package wasn't available on NixOS, so I replaced that functionality with xdotool.
  • I tried to get NetworkManager up and running, and there appear to be configuration options to get it set it up on NixOS, but I ran into issues. I ended up just installing the wpa_gui tool to manage my wireless connections.
I also had a cache of scripts that I used for various things. Some of them were for building programs from source, but I think I can drop those and just use Nix to accomplish the same thing. Others I had to replace '#!/bin/bash' with '#!/usr/bin/env bash' since NixOS doesn't have things at fixed, standard locations like other Linux distributions.

There are some other minor issues to work out. For one my sound card seems to be detected just fine, and Skype will use it, but Xfce doesn't seem to be playing any sounds. I just get a system beep for "sound" events. I think this is an issue with the configuration of Xfce and its sound theme or something. Not really a major issue for me, but I'd like to figure it out eventually.

I would also like to get Dropbox installed. It actually worked fine on 13.10, but for some reason it isn't working so well on unstable. I haven't had a lot of time to look into it, and again not a high priority. I may take the opportunity to look into OwnCloud instead.


I went into this expecting it to be rough at times, and expecting to make some changes to the tools I was used to using. That was no suprise. I was a bit disappointed in how the latest official release of NixOS turned out. The flip side is that the Unstable release has worked great so far, and I'm a little more comfortable following the bleeding edge of NixOS given that (theoretically) I should be able to rollback any changes that bork my system. I'm hoping that turns out to be true.

If you're willing to learn something new and possibly snag your sweater on some rough edges, then NixOS could be a fun experience for you. :) If you use some of the same tools as me, then hopefully what I've discovered can help you move to NixOS. If you want something a little more polished, then you may want to wait on NixOS.


My configuration.nix
Dotfile changes moving from Ubuntu+Gnome+XMonad to NixOS+Xfce+i3