Tuesday, January 14, 2014

The Solution for Silicon Valley?

"People—especially in the financial community—seem to assume that every industry works kind of like a web startup, and that all you need is two hot guys and $25,000 and you're a millionaire in six months. Heavy semi doesn't work like that. Heavy semi is like steel mills and railroads. By the time you can get a serious semi company self sustaining you're looking at a couple hundred million dollars of investment." — Ivan Godard of Out-of-the-Box Computing (https://www.youtube.com/watch?v=uotQn-jrAZU#t=599)

It's a longer conversation to have, but I have opinions about Silicon Valley (that I share with others). However, it is nice to see people trying to create a real business and solve real problems instead of creating the next Instagram clone. The thing is that it takes hard work, many years, and the perspective to step back and work towards fundamental—not incremental—change.

Best of luck to the OOTBC guys! True, a new CPU architecture isn't exactly solving problems of justice and social inequality, but at least it isn't a get rich quick scheme. Finding ways to fill in computation at neglected price points in the market is a stepping stone to solving other problems. Hopefully those problems aren't how to gamify haircuts or something. *sigh*

Let's have some get-rich-slow startups that are truly innovating to solve difficult, fundamental problems.

Wednesday, January 8, 2014

The Buck Stops Here

"Rufer explains the thinking behind the process: 'I was signing checks one day and I recalled the saying, "The buck stops here." I thought to myself, that isn't true. In front of me was a purchase order, a note that said the stuff had been shipped, we had received it, and that the price on the invoice matched the puchase order. A check had been prepared. Now, do I have the choice not to sign the check? Nope. So the question isn't where the buck stops, but where it starts—and it starts with the person who needs the equipment. I shouldn't have to review the purchase order, and the individual shouldn't have to get a manager's approval.'"

"First Let's Fire All the Managers," Gary Hamel (https://archive.harvardbusiness.org/cla/web/pl/product.seam?c=573&i=15715&cs=7c855bfce2fd1c3860846954978b1181)

Monday, December 30, 2013

NixOS: A Field Report


NixOS is a relatively new Linux distribution based on the Nix package manager. A Nix package is hashed based on its content, dependencies, compiler , etc. and stored immutably in the Nix store. The Nix package manager uses this Nix store, a patched dynamic linker, and a buttload of symlinks to isolate installs. You can have multiple versions of applications and libraries living on the same machine without conflict.

The Nix package manager also maintains a history of installations and—since the Nix store is full of immutable installs of applications and libraries—Nix provides a way to rollback installs to a previously working configuration. NixOS extends this concept all the way to the Linux kernel. When you change the system configuration for your machine you get a new GRUB entry for that entire profile of your computer: kernel, applications, libraries, and even your configuration files. If something goes badly you just reboot and select the GRUB entry for your previous configuration. Check out section 1.4 of the NixOS manual for some cool details http://nixos.org/nixos/manual/#sec-upgrading.

But wait there's more! These installations are atomic, so it either completes successfully or not at all. So basically what we get here is something like an ACID MVCC for the software on your computer.

There are definitely practical benefits to all this. There have been times that I've done some riskier upgrades. Moving to a new Ubuntu release is one of those times. There's always a bit of a question as to what may or may not be broken. I usually back up my files and do a fresh install. I've also been bitten a couple of times by a bad video driver upgrade that was annoying to recover from. That said, I don't know that I've really needed an ACID MVCC for software all that often.

Aside from riskier installs you can also setup a Nix configuration for each project you work on. You can chroot into this Nix environment. The Nix configuration can be version controlled, so that when you checkout an old branch you can use the environment that matches that version of your project. You can also take this Nix configuration and create a VM from it, or send it off to an EC2 node. Admittedly you can do all of this stuff just using the Nix package manager on Linux or OSX without needing NixOS. (Here is how http://zef.me/5966/setting-up-development-environments-with-nix.)

So, really...Why?

So all this stuff is great, but maybe not worth switching to NixOS for your day-to-day desktop machine. I have moved for a couple of reasons. Lately I've been getting disillusioned with the direction Ubuntu is headed. I barely made it through the last LTS upgrade. I prefer to use Gnome and XMonad, but Ubuntu comes with Unity these days and it's hard to get a plain Gnome install (though I made it work). The flip side is that Ubuntu is a really smooth install where most everything just works out of the box, but that's not really enough to keep me there. I'm basically forced to consider some kind of a major change whether it is a change in distribution or the tools I use or something.

I also like to try to change things up every once in a while. I've been on Ubuntu for close to 10 years and Gnome+XMonad for maybe 3 years. I need to shake things up a bit, learn some new stuff, and feel like a hopeless newb. Boy did I ever accomplish that!

Attempt 1: No Gnome & Miserable Failure with Wireless

I downloaded and burned the NixOS 13.10 minimal CD. It booted up fine and gives you hints to walk you through the process. The CD boots with a copy of the manual on one of the Linux consoles so you can jump back and forth to reference the manual as you're installing. The process is pretty simple, too. You format a partition, mount it, run a config generation program, and customize the generated config. Once the config is set you run an install command and boot into your new NixOS installation.

However, Gnome doesn't install. Based on a reading of the docs you might be led to think that it does, but after 13.10 was released this commit was made to the Nix packages: https://github.com/NixOS/nixpkgs/commit/a734f32fa1efce21a008e391cfb10695c4d738cd, so it's not even an option in more recent versions of NixOS.

Coincidentally XMonad doesn't install either. Well it seems like it does, and if you use it without any custom config then it's probably fine, but to use a custom config you need a Haskell compiler and the XMonad packages. For some reason when the XMonad packages install they don't install the dependencies that they need (which seems broken given how Nix is supposed to work). I had to iteratively run `ghc-pkg check` and install a bunch of packages manually. Perhaps I was missing something. Also, not to give away the ending, but this was with 13.10, so it may be a different story with the latest unstable NixOS.

Finally, I had wireless issues. The installation CD booted and I was able to get an install, but once I tried to find some combination of CLI and/or GUI software to configure my wireless adapter I wedged something, and decided to just reboot and reinstall, but the installation CD was hanging when starting the WPA supplicant. It seemed maybe there was some state stored in the wireless card that was causing the WPA supplicant to hang, but I couldn't boot NixOS in order to configure the card. After many hours fiddling with this to try to get the wireless working and being very frustrated I very nearly gave up on NixOS. As a last ditch effort I decided to try using the unstable version of NixOS.

Attempt 2: Using Unstable NixOS

Out of the gate Unstable NixOS was more stable than the 13.10 release. I was able to actually boot the thing! I made it through the installation process. I had decided that I was going to use Xfce+i3 instead of Gnome+XMonad. Changing my desktop and window manager at the same time as changing my distribution was going to make this whole process more "interesting."

Configuring NixOS is relatively simple once you know what the configuration options are, but finding them isn't as easy as it could probably be. The NixOS manual has a section of configuration options at the end. That was an important reference. I don't know if it documents every possible option, so sometimes I would take a look at the Nix package source. There are also configuration options that affect some packages that aren't entirely obvious unless you look at the package source. I think in the future it would be nice (in addition to the way the documentation is currently organized) to have documentation that can be browsed per package, so you could see for a particular package what options affect it.

Having a working distribution is since, but there were still some annoyances. Searching for packages is also a bit unsatisfying. The recommended way to search seems to be `nix-env -qaP '*' | grep NEEDLE` which works, but it's not the most performant, and you also can only search by package name; you can't search by description or configuration options or anything else. Even when you've found the package you want more detail on, it is not obvious how to find the package. You can browse the nixpkgs source, but it is organized by categories. The best way seems to be doing a `find / -name NEEDLE` or something. None of this is really a deal breaker (for me), but just lessons learned on how to find information.

Finally, on the subject of finding information the nix-dev mailing list is a good resource. I ended up searching it with google "site:http://lists.science.uu.nl/pipermail/nix-dev/ NEEDLE". I guess it is also possible to search the Gmane archive http://news.gmane.org/gmane.linux.distributions.nixos.


There were definitely tools that I'm used to using that were available and worked great on NixOS, among them were Conkeror, Emacs, OpenJDK, Ruby, Tmux, and Skype.

There were replacements I had to make. Obviously Gnome+XMonad became Xfce+i3, and this had a cascading effect.
  • I had a raise-or-run script that used wmctrl to activate a program's window, but wmctrl doesn't work with i3, so I had to replace that script with one that uses i3msg.
  • I spent a bit of time tuning my i3 config to have bindings similar to the ones I was using with XMonad.
  • Previously I was using gnome-panel for workspaces, wireless control, sound volume control, date/time, etc.; that got replaced by i3bar and i3status.
  • I was using swarp to shuffle the mouse cursor out of the way in Gnome+XMonad, and that package wasn't available on NixOS, so I replaced that functionality with xdotool.
  • I tried to get NetworkManager up and running, and there appear to be configuration options to get it set it up on NixOS, but I ran into issues. I ended up just installing the wpa_gui tool to manage my wireless connections.
I also had a cache of scripts that I used for various things. Some of them were for building programs from source, but I think I can drop those and just use Nix to accomplish the same thing. Others I had to replace '#!/bin/bash' with '#!/usr/bin/env bash' since NixOS doesn't have things at fixed, standard locations like other Linux distributions.

There are some other minor issues to work out. For one my sound card seems to be detected just fine, and Skype will use it, but Xfce doesn't seem to be playing any sounds. I just get a system beep for "sound" events. I think this is an issue with the configuration of Xfce and its sound theme or something. Not really a major issue for me, but I'd like to figure it out eventually.

I would also like to get Dropbox installed. It actually worked fine on 13.10, but for some reason it isn't working so well on unstable. I haven't had a lot of time to look into it, and again not a high priority. I may take the opportunity to look into OwnCloud instead.


I went into this expecting it to be rough at times, and expecting to make some changes to the tools I was used to using. That was no suprise. I was a bit disappointed in how the latest official release of NixOS turned out. The flip side is that the Unstable release has worked great so far, and I'm a little more comfortable following the bleeding edge of NixOS given that (theoretically) I should be able to rollback any changes that bork my system. I'm hoping that turns out to be true.

If you're willing to learn something new and possibly snag your sweater on some rough edges, then NixOS could be a fun experience for you. :) If you use some of the same tools as me, then hopefully what I've discovered can help you move to NixOS. If you want something a little more polished, then you may want to wait on NixOS.


My configuration.nix
Dotfile changes moving from Ubuntu+Gnome+XMonad to NixOS+Xfce+i3

Sunday, December 22, 2013

A Proof of the Braininess of "The Simpsons" and "Futurama"

Occasionally the mathematics does wind more deeply into the story, most notably in the 2010 “Futurama” episode “The Prisoner of Benda.” The plot turns on a device called the Mind-Switcher, which performs just the function its name suggests; over the course of the episode, as the apparatus is used with greater and greater abandon, the minds of the characters shuttle from body to body like singles switching bedrooms in a French farce. By the end, not a single consciousness remains in its proper skull. What’s worse, the characters can’t just retrace their steps to reunite each mind with its original body; the Mind-Switcher, having operated on a pair of minds, isn’t allowed to switch the same two minds again.

Ken Keeler, who wrote the episode, realized that in order to get everything sorted out it might be necessary to introduce new characters, whose bodies could be used as waystations through which the minds could find their way home. An ordinary writer would have been content simply to find a way out of the episode. But Keeler became obsessed with the problem in its full generality, finally composing a proof that, no matter how wild the original fiesta of Mind-Switching, the damage can always be repaired once two new people are added to the system. This question may sound abstruse, but the part of math it belongs to — “combinatorial group theory” — is one of the hottest things going at the moment, with major advances popping up everywhere from Paris to Los Angeles. Keeler’s theorem isn’t one of those big advances, but it’s a real theorem, certainly the deepest piece of mathematics ever featured in a prime-time sitcom.

-- Jordan Ellenberg, "Mathematics and Homer Simpson" (http://www.washingtonpost.com/opinions/2013/12/20/ef1bfaa6-5b9a-11e3-bf7e-f567ee61ae21_print.html)

Monday, December 16, 2013

The Notion of "Real-Time"

"I think the current evolution of technical language around web developers has made real-time mean: 'consume information as soon as it is available' and not '_react to the information in a timely manner or this car will crash_'."

-- Alvaro Videla, "Tell me more about your real-time systems" (http://videlalvaro.github.io/2013/12/tell-me-more-about-realtime.html)

Wednesday, July 31, 2013

Remote != Distributed

In an age when companies seem to be shutting down their remote work programs, I'm here to tell you that a distributed team can work, and when it works well it is a great experience. I have worked on a distributed team for the past three years. Prior to that I worked for four years in an office that had one full-time remote employee and allowed occasional work from home for others.

Working from home and working remotely are not the same thing as working on a distributed team. If there is a center of gravity at an office and a few remote employees orbit that office, then there is a different experience for some people than for others, and there is a different way of working for some people than for others. There is a natural imbalance that an inanimate system would want to equalize, but in a human organization people pull in opposite directions.

A distributed team is a team where no one is physically co-located. There is no center of gravity at an office. There is no imbalance, because everyone has the same experience, and everyone has the same way of working.

When there is an office/remote split, there is a different experience for some than for others. The office people have shared experiences like water cooler chat and birthday celebrations, and the remote people are left out. The office people gather in a conference room for a meeting and dial in the remote people over a crappy speakerphone. The experiences are different which affects morale and cohesion.

On a distributed team everyone has the same experience. You find new ways to create shared experiences that are different from office experiences and unique to distributed teams. You use tools that allow meetings to be high quality experiences for everyone. These shared experiences support morale and cohesion.

An office/remote split means different ways of working for different people. In-office employees may depend on ad hoc hallway conversations, but remote employees are not privy to those. In-office employees have an "out of sight, out of mind" mentality and think that remote employees aren't productive because they aren't communicating with the remote employees. Remote employees feel disconnected and lack direction for the same reasons.

On a distributed team everyone communicates through the same channels, and everyone can see the activity about who is doing what. Since there aren't two different communication channels there isn't an "out of sight, out of mind" problem.

When there are different experiences, different ways of working, different communication channels at the same company, then two different cultures develop, and that is detrimental. A permanent office/remote split is doomed to failure from the beginning, so it's no wonder that companies are killing their remote work programs.

What does it look like to do a distributed team right? I can tell you what has worked for me and my company. I do not expect that our experiences are universal, but I will try to draw some general principles.

Distributed teams that work well:
  • Use tools to fix work into a tangible medium of expression (to borrow from copyright law). Office collaboration via ad hoc hallway conversations and whiteboards doesn't work for distributed teams. You have to produce Google Docs, tickets in Jira, etc. These are things to which everyone can have access. Frankly, distributed or not, this is a better way of working than informal ideas banging around in people's heads.
  • Communicate through many-to-many channels. Of course there is still a need for one-to-one communication, but using many-to-many communication channels means that everyone can feel a part of the company. Just like others can walk by and join in on hallway conversations, people can observe and contribute to Skype calls and chat rooms. It is also a good idea to have logs of these chats that people can catch up on (see above).
  • Adopt processes that encourage collaboration and productivity. Working on a distributed team may not be for everyone; to a certain extent it does take a self-motivated individual. However, pair programming and daily standups create an environment that encourages collaboration and accountability. We also tend to hire people mostly in North and South America so that the timezones line up better for collaboration. We also meet face-to-face 3-4 times a year, which helps to develop the personal relationships that are necessary to working well together.
Realize that the idea of "remote" working does not work when combined with an office culture and a center of gravity in one location. Working from home or remotely may work temporarily, but eventually gravity will pull you back towards the office. If you try to maintain this unnatural office/remote balance you will enforce different experiences and methods of work and create two different cultures in your company. This cultural divide will lead to jealousy, resentment and other morale and cohesion problems.

This does not mean that it is impossible to function unless everyone reports to the same office. It just means that different tools and processes must be developed to facilitate a distributed team. You must use tools and processes that encourage tangible expression of ideas, many-to-many communication, and collaboration and productivity.

Remote working is isolating and doomed to failure, but a distributed team done right is a joy.

Wednesday, November 28, 2012

Evaluating CLEAR Wireless Internet

Here's my current situation: I have Cox High Speed Internet at home in Fairfax County, Virginia, and I have a Verizon Wireless smart phone with the HotSpot feature enabled. I've been intrigued by Clearwire for a while, and I'm always on the hunt for improving my Internet access at home and on the road. Though I've been fairly happy with HotSpot access through Verizon (though not so happy with my HTC Rezound, but that's a story for another time), Clear has a 15-day money back guarantee, and no contract, so it seemed like a no brainer to try them out. Also, I checked their coverage and I should get a good signal at my house.

I think in an ideal world I'd like to decouple my smart phone and my mobile Internet. I'd like to have a smart phone with minutes and texting, and connect it to the Internet (as well as other devices) through a mobile HotSpot, like Clear. However, it doesn't look like that plan is going to come to fruition any time soon. For one thing, Verizon won't allow me to have a smart phone on their network without some kind of data plan.

How does Clear stack up?

The experience signing up for and using Clear has been really great. (I'll let you know how it goes returning the device and getting my money back...but I'm getting ahead of myself.)

I signed up on their website, and received the device the next day. I plugged in the device and it connected to the 4G network. I was able to connect my computer to the HotSpot device without any problems at all.

I tested the "4G Internet Basic" plan which has an advertized speed of 1.5Mbps down and 500Kbps up for $34.99 per month. According to speedtest.net I was getting 1.92Mbps down and 460Kbps up with a ping of 72ms.

It looks like you get what you pay for, and the experience is very smooth, but for me the price/speed isn't as good as I can get elsewhere. The HotSpot on my smart phone gives me 9.93Mbps down and 3.21Mbps up with a ping of 58ms for $30 on top of my unlimited data plan. (I'm grandfathered in on the Verizon plan I'm on. I'm not sure what the numbers would look like if I were to sign up today.) And my Cox High Speed Internet gives me 39.48Mbps down and 12.33Mbps up with a ping of 18ms for $40 per month.

Clear has a "4G Internet Plan" which has an advertized speed of 6Mbps down and 1Mbps up for $49.99 per month, but that is getting on the steep side for me, and the analysis doesn't help Clear.

Comparing my options, $1 will purchase 1Mbps from Cox, 0.333Mbps from Verizon, 0.059Mbps from the 1.5Mbps Clear plan, and 0.122Mbps from the 6Mbps Clear plan.


I would really like to use a service like Clear, and I have no complaints about the Clear experience. However, it is not the best value for me. If Clear was charging $6 per month for their 6Mbps plan, I'd be all over it, maybe even if they charged $10 per month for their 6Mbps plan. However, as it is, I'm going to pass.