Thoughts on the Death of Brook's Law

Brooks' Law is wrong.

Brooks' Law (the pithy version) is: "adding people to a late project makes it later." There is some truth to that, but it's more nuanced. A bad manager staffs up a late project (in an act of desperation?) to make a deadline. However, Bertrand Meyer (and Steve McConnell and Barry Boehm) believe—based on empirical evidence—that judiciously adding people to a project can shorten its schedule.

You can shorten a schedule by adding more people...however there's a limit. You can only shorten the schedule for a software project by (up to) 25%. Since adding people to a software project means adding cost, what this is really saying is you can spend more to get your thing up to 25% more quickly. However, the reverse is not necessarily true. Sometimes you can take people off a project, give the remaining people more time, and get your thing for less money; sometimes you cannot.

This has been known for about 20 years and yet Brooks' Law has survived as a folk wisdom, and Bertrand Meyer wants to get the word out. That's why he wrote "In Search of the Shortest Possible Schedule."

I grew up on Brooks' Law, so I'm trying to absorb this. Brooks' Law seems right to me, and in "Brooks' Law Repealed?" Steve McConnell describes an experience that seems familiar:

To those of us who have been around the software-project block a few times, the claim feels true. We’ve participated on projects in which new people are brought on at the end. We know the irritation of having to answer questions from new staff when we’re already feeling overwhelmed about our own work. We’ve seen new hires make mistakes that set the whole project back. And we’ve experienced additional schedule slips even after staff has been added to a late project.

McConnell squares this experience with the new sans Brooks' Law world by pointing out Brooks' Law does apply in certain circumstances, but projects are poorly estimated and poorly tracked. The result is not knowing whether you are in the Brooks' Zone or whether there is enough project left for new hires to pay off the productivity lost to training them.

I'm not sure I buy that. I think the idea of estimation is fundamentally flawed. I can't help but feel like this is saying, "We're doing a bad job. Do better!" I'm more and more convinced that breaking work down, estimating the pieces, and rolling it back up is a terrible way to estimate. It fails to account for variability, and padding estimates is not the solution.

And sure, better project tracking seems like a good thing. It is a necessary first step, but having the data isn't enough, you also have to interpret and extrapolate it. Probably the best thing that can be done with tracking data is to let it empirically drive estimations.

I cannot deny the empirical evidence. You can pay more to shorten a project by up to 25%, but there are some intriguing questions that pop up: Why? Why 25%? Why doesn't it always work backwards? Could you repeat the process with a revised schedule and cut another 25%? How would knowing about this bias estimations?

I think what I take away from this is that Brooks' Law has narrower application than I initially expected and adding people to a project can bring it to completion more quickly.

On Writing (Code)

Introspection into my own process for making software has led me to believe that writing prose and programming are fundamentally the same. This was further highlighted for me when I read Stephen King's On Writing. I saw interesting parallels between the way he likes to work and the way I like to work. Also my code is a horror story.

King likes to write a first draft as quickly as possible. He does this to get the story out. He doesn't worry about character development or even holes in the plot. Those he will fix up in the second draft. The first draft is about getting onto paper the good bones of a story. Then he lets the first draft sit for a couple of weeks.

When he can come back to the first drift and it looks familiar but not quite—like it was written by his doppelganger—then he has enough distance to make the second draft. That's where any story issues are fixed and things are smoothed out and tightened up. His goal is to make the story 10% shorter.

When I write code:

• I like to write code breadth-first so I can see all the moving parts and that everything fits together.
• I like to let it sit for a couple of minutes, or hours, or days—whatever I can afford.
• I like to smooth things out and tighten things up in the second draft—things like function names, refactoring, etc.

Maybe writing code isn't exactly like writing prose. Maybe it's more like knitting or woodworking or cycling or solving crimes. I think part of the reason that programmers tend to see their work in everything, is that programming is really just process for solving problems and giving form to thoughts. Those skills can be applied to many different domains. Maybe in that way it actually is more like writing than other hobbies?

At least I didn't say that programming is like gardening. I actually appreciate gardening as a hobby because of the fact that it is entirely different than making software!

How to Survive Life Undamaged

How does one survive life undamaged? Seriously... if you figure it out let me know. I particularly mean self-inflicted damage. You can't guarantee anything about how other people act or think.

Here is what I have tried (am trying):

1. Be willing to listen to others.

You don't have to listen to everyone, but be aware that you risk losing valuable perspective and insight, even if it is insight about how not to do things. You should probably be willing to listen to some people that you have shut out. If you are not uncomfortable, then you are not growing.

Everyone has a story. You can either imagine a person's story, categorize and label them, and treat them as if that story is true, or you can actually listen. You can listen to what someone believes while thinking about why they're wrong and how you will show them, or you can try to inhabit their space and see the world the way they see it. You don't have to stay there, but you will take lessons from it.

2. Be skeptical by default.

To listen to others and inhabit their space does not mean uncritically accepting their ideas. You can be empathetic, you can entertain an idea, without being swept away. You can also question if a person is wrong without imagining they are your enemy.

If some idea pushes through your skepticism, accept it. It is OK to move a little closer towards an "opposite" view.

3. Don't try to convince everyone.

This can be particularly painful when it is someone you respect and/or love. It may take time. Or it may never be. Life is sour sometimes. Don't make it more sour than it needs to be.

If you are skeptical by default, then maybe others are as well? If you are listening to others and inhabiting their space, then you must know that it will not be easy to convince everyone. You can let that bother you. You can turn it into an obsession, or you can focus on the good (what little you estimate there to be) and seek friendship.

These are three things you can try. If you refuse to listen to others and make it your life goal to convince everyone how they're wrong. You will have a life filled with stress and bitterness. Or you could listen--critically--and seek friendship over rightness.

Beyond Techniques

I firmly believe that writing software is a creative act. I believe that a construction project---or any other linerally presenting process---is the wrong analogy for software development. A much better one is writing prose, which is a process of writing and rewriting and rewriting, and sometimes throwing it all out and starting over. In the same way a software engineer names and renames, factors and refactors, etc., then extends the functionality by doing it all again. This all has to fit into the larger code base in a way that makes for a consistent whole. This requires judgement and taste.

To come at the argument from the other side, if you spent your days rewriting the same code over and over, then you would split it into more generic functions and perhaps a library. If you're solving the same problem that someone else has solved, then you would just reuse their software. This does in fact happen, and yet software engineers still get paid to work, and the value they generate is in the new stuff that requires creativity. Sure, there are some inefficiencies that mean people rewrite software that they could have just reused, and you could argue that the new things to be done are increasingly marginal, but on the whole software engineers would be out of work if there were not new things to create. Ergo writing software is a creative act.

The next reasonable question is: how does one create? I believe that fundamentally, and on average, one cannot directly induce creativity. This isn't entirely satisfying. I wish creativity was a predictable, mechanical process. Is it possible to discover and categorize tools for thinking? I'm still exploring that. Here's an example: to get a fresh perspective on your problem think not about how to solve it, but how to avoid failing to solve it. Instead of trying to figure out how to build a stronger bridge, maybe you'll gain insight by considering how to avoid building a weaker bridge. This technique for finding fresh perspective...does it not make creating into a predictable, mechanical process?

I actually like techniques. I believe the learning process is (or is best) technique based. Think about learning to cook. You start by following the recipe. Then you start to improvise. You understand that this ingredient is for flavor and can be adjusted to taste, but that ingredient provides aeration and messing with it too much will produce a dense, inedible result. Eventually you move beyond techniques and start to construct your own recipes from first principles. A poor education either starts with first principles, which a beginner does not have the experience to appreciate, or focuses too much on techniques divorced from context, which frustrates learners because they don't know how to properly apply techniques. I intended my book Clojure Polymorphism to position itself as a way to explore varied applications of the same tools and techniques, which I hope will guide readers from knowledge into wisdom.

The writing curriculum we use in homeschooling our children is technique based. A child reads an essay and creates an outline of the important points (there's a whole set of techniques for that). Then he will rewrite the essay from the outline applying techniques like "ly"-words, openers, and clinchers. This removes the hand wringing about what to write about. (Incidentally, if you read Ben Franklin's autobiography, this is the way he improved his own writing.) What if computer programming was taught this way? Here's a program that consumes a CSV file, changes some things, then writes it back out. First write out a high level algorithm for the program, then rewrite it using 3 of the 15 techniques you've learned (concurrency, multimethods, asynchronous channels, etc.). Repeat. I think this would produce wiser programmers.

However, eventually you need to move beyond techniques. Leaning on techniques is just shifting the problem elsewhere. Coining techniques is an attempt at mechanizing creativity, but instead of making creating mechanical, you are now faced with choosing which technique to apply to generate a creative solution. Now the creative leap must occur earlier in the process. You can try each technique in a brute force search, but I doubt many (if any) would consider that "creative," nor is it particularly efficient. The experience of repeatedly applying techniques should help you develop judgement and taste, so when you're faced with inventing from first principles, you have some gut sense or guiding aesthetic. That guiding aesthetic is not some set of rules for when to apply which techniques. If it were, then it would become a "library," so to speak. Anybody---or any machine--could evaluate the rules and perform the steps.

Techniques are great for broadening your learning, but they are not self applying. Learning all the techniques does not help you invent new techniques, nor will it induce creativity. You can know how to cut impeccable dovetails and square up lumber and drill straight holes, but that does not mean you will create beautiful furniture.

How does one develop this guiding aesthetic? You read a lot of other people's code. You write a lot of code. You read a lot of books. You work hard. Sorry, that's the best I've got. This is a continuing journey...to be continued.

However you do it, the end goal should be to move beyond techniques.

Virtual Machine Oriented Development

Most computing devices that we have today---desktop, laptop or phone---are capable of computing any program that can be computed. There's a bit of equivocation there. What is meant is that anything that a human can manually calculate via rote, mechanical process can also be done by computer. This is the Church-Turing Thesis.

I've never really stopped to think, but what would a non-universal computing machine look like?

....

Longevity

Several years back I suffered a bout of jealousy. I thought about engineers in other fields who build roads or buildings or even cars. A civil engineer can imagine something they've build standing still amidst the blur of 100 years passing by. An automotive engineer can imagine a car they've designed still driving the roads in 20 years.

I don't think a single line of code that I wrote 3 or more years ago is still in production, and that's ignoring all the code that I wrote that never made it into production.

These are the kinds of things you start to ponder as you reach the ripe, old, programmer retirement age of 33.

But then a funny thing happened. I played Sam & Max Hit the Road ... on my Android phone.

Here was a game released in 1995 and I was playing it on my Android device in 2014. How did that happen? Well, when LucasArts designed the game "Maniac Mansion," they decided to create a scripting language and write the game in that language, and they used that scripting language for many of the games they made. I have several original LucasArts games on CD. Some are PC versions some are Mac versions.

Over the years as I feel the nostalgia hitting me I'll grab the game files from the CD and download ScummVM for whatever platform I'm on at the time. I copied the game files to my phone and downloaded ScummVM from the Play store. That's how it happened.

....

Data is Code

I had been exposed to Lisp, and even written a lot of Lisp before I finally had my enlightenment about macros and metacircular interpreters. I remember vividly reading Structure and Interpretation of Computer Programs and seeing Scheme put to use creating simple yet powerful abstract interpreters. The author's start "simply" with interpreters that add new programming paradigms to Scheme. Then they proceed to simulating a register machine and writing an assembler and compiler for it. This happens in the last chapter, a space of ~100 pages.

It is a Divine joke that Structure and Interpretation of Computer Programs and The Art of Computer Programming had their titles swapped, because---while I don't wish to denigrate TAOCP which is a depth of amazing riches---SICP is about art, and in a metacircular way it is art.

It is too easy as a Lisper to understand the world this way, but data is always code. In Forth, 5 is not a number, it is an instruction to push the value 5 onto the top of the program stack. Your program receives a program as input. It receives files, network packets, key presses, and mouse clicks. It interprets this program and produces output.

A PDF file can cause a buffer overrun in a PDF reader because each byte is literally an instruction to your program-as-interpreter to "write a value at the current location and move to the next location" (or at least it can be if your program-as-interpreter has flawed semantics).

This is not a property of Lisp, it is a property of the stored program computer, Universal Turing Machine, von Neumann architecture. Code and data are made of the same stuff and stored in the same memory.

....

The Non-Divine Joke

In his talk "The Birth & Death of JavaScript," the 2014 version of Gary Bernhardt extrapolates where JavaScript and asm.js will take the world in 2035 (after an apocalyptic global war, of course). The punch line is that instead of JavaScript running on computers, computers run on JavaScript. This happens through a comical stack of emulators emulating emulators that emulate. Actually I think it's compilers transpiling compilers that transpile transpilers, but...same difference.

But like every joke there's a bit of truth to it.

Paul Graham writes about "Programming Bottom-Up" where you build the language "up" to your program to the point that actually expressing your program becomes somewhat trivial. You're building a domain specific language to solve exactly the problem you have. Again, this is all too natural for Lispers, but everyone does it.

The act of programming is to turn a universal computing machine into a limited computing machine. You build out data types and operations to focus the abilities of the computer into a specific domain. Programmers instinctively understand this, which is why we find it so funny that---in a twist of irony---a universal computing machine emulates a universal computing machine emulating a universal computing machine.

....

Virtual Machine Oriented Development

I started thinking about Virtual Machine Oriented Development because I was concerned about the transience of my legacy. I noticed that there were software products that were still around 20 years after they were written. I started seeing a VM underneath them.

But having thought about it more, I don't think that Virtual Machine Oriented Development is just about legacy. I think it might clarify the design process to be explicit about the fact that we're designing a limited computing machine that analyzes sales data. What are the data types? What are the operations? If you have power users, maybe they'd even like a scripting language that can describe which data to import and then how to analyze it?

You might find then that you've abstracted your problem into a computation model that will become valuable for years. Maybe you'll end up rewriting the interpreter for this language several times, and all the while users can keep using their existing scripts.

....

Conclusion

What does a non-universal computing machine look like? It looks like every program you've ever written.

Speed Reading

My reading habits are lumpy. I find I'm either not reading anything, or I'm reading seven books at once, and when I am, I wish I could speed read. I have in the past read books on speed reading, and they usually boil down to techniques like increasing your eye span, eliminating regression, eliminating subvocalization, etc. The theory seems to be that your brain can work much faster than your eyes, and you just need to eliminate bad habits, and establish some better ones, so you can get the words into your brain faster.

I do feel like there's something to this. In my experience, my mind tends to wander as I'm reading, and sometimes it's easier to skim since that keeps my mind busier trying to assemble random bits and pieces into a comprehensive whole. That seems to be the idea with this new speed reading book that I picked up called "Speed Reading With The Right Brain." The author claims that by engaging your brain in conceptualizing what you're reading as you're reading, you increase comprehension, and it is through increased speed of comprehension that you achieve increased reading speed. I want to believe, but I'm still skeptical.

What I would love is some reference that approaches speed reading from an empirical approach, you know, with science. What do we know that actually works based on research? Well...you may not like the answer.

Speed Reading is Fake

In looking for an empirically backed approach to speed reading, I came across "So Much to Read, So Little Time: How Do We Read, and Can Speed Reading Help?" This article is based on decades of reading research and cognitive psychology. One of the authors—who passed away from cancer a few days after the first draft—proposed the article "because he felt that it was important to share the knowledge we have gained from experimental science with the general public."

While there may be savants who can read impossibly fast without sacrificing comprehension, controlled studies show that for normal people learning to read faster means comprehending less. When you learn to "speed read" you are actually learning to skim. "Taylor notes that [Evelyn] Wood 'repeatedly stated that her people are not skimming, but rather are reading' (Taylor, 1962, p. 65). Based on recordings of their eye movements, however, Taylor concluded that they closely resembled the eye movement patterns produced during skimming (Taylor, 1965; see also Walton, 1957)." Also from the article:

The speed readers did better than skimmers on general comprehension questions about the gist of the passages but not quite as well as people reading at normal speed. … The advantage of trained speed readers over skimmers with respect to general comprehension of the text was ascribed by Just and colleagues to an improvement in what they called extended inferencing. Essentially, the speed readers had increased their ability to construct reasonably accurate inferences about text content on the basis of partial information and their preexisting knowledge. In fact, when the three groups of participants were given more technical texts (taken from Scientific American), for which background knowledge would be very sparse, the speed readers no longer showed an advantage over the skimmers, even on general questions.

According to the article, learning to speed read may improve your ability to skim, but only for familiar subjects. This isn't necessarily bad news for two reasons:

1. You can improve your reading speed, just not as dramatically as speed reading advocates suggest.
2. Learning to skim effectively is a useful skill to learn.

Improve Your Reading Speed

The average person reads between 200-400 words per minute. What is the best way to improve your reading speed? Practice. Unsurprising. Perhaps a little disappointing? There are a couple ways that practice increases your reading speed, but they basically break down to improving your language skills:

• better vocabulary
• exposure to more writing styles

The broader your vocabulary, the more familiar you are with words and styles, the more quickly you can read. Your eyes fixate less on words with which you are familiar, so they move more briskly across the page. Your familiarity with style allows you to anticipate better how a sentence will end when you've only read part of it. Also, "written language uses some vocabulary and syntactic structures that are not commonly found in speech, and practice with reading can give people practice with these."

The more you do reading, the faster you'll get.[1]

Learn to Skim Effectively

Effective skimming is mostly about trying to extract structure and important ideas from a text. Scan for:

• headings
• paragraph structure
• key words

According to the article, "Research has shown that readers who pay more attention to headings write the most accurate text summaries (Hyönä, Lorch, & Kaakinen, 2002)."[2] You can also do things like:

• scan the table of contents
• read the first paragraph of each section
• read the first sentence of each paragraph

Again from the article:

The eye movements revealed that skimmers tended to spend more time reading earlier paragraphs and earlier pages, suggesting that they used the initial parts of the text to obtain the general topic of passages and provide context for the later parts that they skimmed in a more cursory way. Therefore, effective skimming means making sensible decisions about which parts of a text to select for more careful reading when faced with time pressure. In fact, Wilkinson, Reader, and Payne (2012) found that, when forced to skim, readers tended to select texts that were less demanding, presumably because they would be able to derive more information from such texts when skimming. This kind of information foraging is a useful method of handling large amounts of text in a timely manner.

You know what else helps you skim effectively? Practice. Practice gives you a broader base of knowledge and experience to draw on:

That [knowledge/experience] may be the basis for some anecdotes about the speed-reading abilities of famous people, such as that President Kennedy could pick up a copy of the Washington Post or the New York Times and read it from front to back in a few minutes. However, consider the knowledge and information that someone like Kennedy would bring to the task of reading the newspaper. As president, he was briefed about important world events each day and was involved in generating much of the policy and events reported in the newspaper; thus, he probably had first-hand knowledge of much of what was described. In contrast, the average person would come to such a situation with very few facts at his or her disposal and would probably have to read an article rather carefully in order to completely understand it. To read rapidly, you need to know enough about a topic to fit the new information immediately into what you already know and to make inferences.

Of course, the downside of skimming is that you are skipping over portions of text resulting in lower comprehension. However, if you're looking to get a general overview or find one specific fact, then it can be useful. It may also be good for a first pass at a text before reading in depth.

Conclusion

I do feel as though my mind wanders when I read, and I wonder whether there is a way to better engage my mind when reading. Perhaps conceptualizing or visualizing or some other way of focusing more would help comprehension and speed. If and until I and figure that out, I can improve my reading speed by practicing and getting better at skimming, when skimming makes sense.

Footnotes:
[1] I wonder (off-the-cuff, anecdotally, non-scientifically) whether writing more also improves reading, for the same reason: it improves language skills.

[2] Interestingly, this means that as an author you bear some of the burden for helping readers quickly consume your writing. I had started to shy away from listicle style blog posts, thinking I'd try to contribute to a more high minded discourse that rewarded effort in reading and comprehending. This article has more headings and lists...maybe I'll do a little of both. :)

Engineering Synthesis

What is the nature of software engineering? How is it different from other kinds of engineering? Why is it so hard?

These are questions I have struggled with for many years. In my work, I have seen more than a few different takes on software engineering. Even when things start out right they seem to end at a sad place, and this has bothered me. Is it really impossible to do software "right?" Or do we just have the wrong idea about how to do it? Software engineering is a relatively new discipline, so maybe we still have some things to learn.

I'm going to draw from several sources here, and try to synthesize some ideas about engineering, science, and art. I feel kind of silly writing all these words summarizing other sources when you could just go watch the videos and read the papers yourself. But for my purposes these sources are a framework for discussing and organizing my thoughts.

Real Software Engineering

"Real Software Engineering" by Glenn Vanderberg
http://www.infoq.com/presentations/Software-Engineering

Glenn Vanderberg is a software practitioner, and he is reacting to the claim that software engineering needs to grow up and become a "real" engineering discipline. But what is "real" engineering?

There are actually a couple of different versions of this talk available online, and in one Vanderberg takes some time to talk about "how did we get here?" He digs up some history on the NATO conference in 1968 whose goal was to define software engineering. He then talks about some commonly believed myths about engineering, about how different engineering disciplines use different methods, then brings it back around to software engineering and applies what we've learned.

There were three big ideas from Vanderberg's talk that stood out to me:

1. The model of scientists discovering knowledge and engineers then applying that knowledge is wrong.
2. Software engineering is unique because we spend a lot of time crafting design documents and models and a trivial amount of time actually producing the end product, which is the exact opposite of most other branches of engineering.
3. Agile methods are the best methods we have and for all practical purposes they are software engineering.

When I first watched Vanderberg's talk years ago, the big idea was the second—about the uniqueness of software engineering—but coming back to it later I was surprised to find this first idea echoed in other sources. Vanderberg gives a examples of advances in knowledge that came not from academics or scientists, but instead from practitioners and engineers. One example is Robert Maillart. He was an engineer who revolutionized the use of reinforced concrete in bridge building. He did this before there were mathematical models to explain the uses and limits of reinforced concrete. Scientific advances are just as likely to come from practitioners as from academics.

My second idea from Vanderberg is that among the kinds of engineering, software engineering has some unique characteristics. If one were to build a skyscraper, one would construct designs, models, blueprints, then those would be handed over to a construction team who would construct the building. The blueprints are relatively cheap to produce. The actual construction is error prone and requires a lot of materials and labor. Looking at this process, it would seem very important to focus as much effort on the architecting of blueprints as possible. Once you've laid the foundation, it is expensive to rethink the footprint of the building.

If I were to apply this process to software engineering I might do something like the following: Hire a system architect to create a design document, and then get a bunch of code monkeys to actually construct the system by writing code. In my interpretation, the requirements and design document are the model and blueprints, the system architect is the architect, and the code monkeys are the construction crew. Vanderberg picked up an insight from Jack Reeves in the 90's: this interpretation is wrong.

Customers do not pay for code, they pay for an executable. They want a working system. That is the constructed product, and it is the compiler not the code monkeys that produces it. The code is the design document and mathematical model. The code monkeys are not the construction crew, they are the architects. Source code and its type systems are a mathematical model that can be formally verified. Using a compiler, I can produce a prototype from that model instantaneously and for free. The source code also contains documentation, and to the extent that it has automated tests (also written in the same language) it is self verifying. Modern high level languages and domain specific languages can even be mostly understood by domain experts.

Software engineering is a unique engineering discipline, because source code is a unique artifact. We should be careful not to take engineering methods from a discipline where constructing a prototype is time consuming and expensive and one is necessarily forced to spend more time on up front design to avoid that cost. This will lead nicely into my third big idea, that agile methods are for all practical purposes the best kind of software engineering we know.

When I say agile methods, I mean agile with a little 'a'. I'm thinking (vaguely) of an incremental tinkering approach, versus a straight line mechanical approach. I'm thinking of a technician approach, versus a technique approach. Or as the original Agile Manifesto said, "people over process." I think they got that right. What is interesting is they were not the only ones to get it right. The original NATO conference on software engineering (1968!) had it right before they had it wrong.

There were two NATO conferences that were a year apart. At the first session Alan Perlis summarized the discussion on system design:

1. A software system can best be designed if the testing is interlaced with the designing instead of being used after the design.
2. A simulation which matches the requirements contains the control which organizes the design of the system.
3. Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself. I think that it is the key of the approach that has been suggested, that there is no such question as testing things after the fact with simulation models, but that in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulation model controlling, as it were, the place and order in which these things are done.

What he is saying is:

1. Test early, test often.
2. Take a breadth first approach mocking out what you need so you can get a sense for the overall system.
3. Iteratively refine the system and replace the mocks.

That is suspiciously similar to an incremental development method. Between the 1968 NATO conference and the 1969 NATO conference things changed, and there was a clear tension between those who thought programming was best done by an expert technician, and those who thought programming was best done mechanistically by someone taught a body of scientific techniques. At the end of the 1969 conference, Tom Simpson gave a talk called "Masterpiece Engineering" which is oozing with conflicts of technician vs. technique.

There was definitely a lot of political maneuvering at the NATO conferences. There are some other resources you can investigate if you'd like. The point is the seeds of agile were there, but for some reason we ended up with 33 years of waterfall.

Engineering(,) A Path to Science

"Engineering(,) A Path to Science" by Richard P. Gabriel
http://www.infoq.com/presentations/Mixin-based-Inheritance

"Structure of a Programming Language Revolution" by Richard P. Gabriel
http://dreamsongs.com/Files/Incommensurability.pdf

Richard Gabriel's talk comes from an interesting perspective. He was involved in the Lisp community and has an academic background (he earned a PhD), but is not an academic. After working as a practitioner, he went back to school to earn a Masters of Fine Arts. Upon returning to the technical community, he felt a paradigm shift had happened while he was gone. The conferences he used to attend had been renamed and were now focused on academics instead of practitioners. His entire field--Lisp systems engineering--and its journals had been deleted.

Then he was given the first scientific paper on mix-in inheritance. Being familiar with previous work done on Lisp based inheritance systems, he felt that this paper was using the same terms to describe some of the mechanisms from the Common Lisp Object System, but the terms had different meaning. Gabriel felt he was experiencing incommensurability, that a paradigm shift had happened from an engineering focus to a scientific focus, and now "scientific" papers were being written that described, as new, things that engineers had already known, using the same terms but with different meanings.

The talk is definitely worth watching. It is an interesting personal story intertwined with technical discussions of the previous work versus the paper he had been given. It is an exploration of whether incommensurability can actually happen and to what extent. He also challenges the myth that science always precedes engineering.

I'm honestly not sure whether Gabriel intended his talk and paper to have a single point. Maybe he is mostly interested in relating his personal experience, but this is what I took away:

1. In general, science does not always precede engineering, and in particular the relationship between computer science and software engineering is even more complex, because the engineers literally create the reality that the scientists study.
2. There are two approaches to software: the systems approach, and the language approach.
3. Making engineering subservient to science means throwing away the progress that engineers can and do make.

This was actually the first talk that started the wheels turning for me on the relationship between science and engineering. I had been told in college that scientists expand the body of knowledge and engineers apply that body of knowledge. Gabriel uses as his example the steam engine. When the steam engine was invented the popular theory used to explain its operation was the Caloric Theory of heat, which stated that there was an invisible, weightless, odorless gas called "caloric" that permeated the Universe. The amount of caloric in the Universe is constant, and its interaction with air molecules can explain heat and radiation, and from it you can deduce most of the gas laws. The Caloric Theory was a useful theory with predictive power. When Laplace adjusted Newton's pulse equations to account for caloric, he was able to more accurately predict the speed of sound.

Eventually the Caloric Theory was replaced by Thermodynamics, and amazingly steam engines continued to work! The steam engine was developed by mechanics who observed the relationship between pressure, volume, and temperature. Whether its operation was explained by the Caloric Theory or Thermodynamics made no difference to them. Yet, an engineer's invention can and does spark the curiosity of a scientist to develop a theory to explain how it is that an invention works. This is even more true in the case of computer software.

The second moral I drew from Gabriel's talk is that there are (at least) two approaches to software: a systems approach and a language approach. Gabriel acknowledges that at first he thought the incommensurability that he saw was a difference between an engineering paradigm and a scientific paradigm, but eventually he saw it as a more technically focused conflict between a systems paradigm and a language paradigm. Perhaps what Gabriel means is that you can approach either systems or languages from an engineering or a scientific perspective. However, I tend to see systems versus languages as engineering versus science.

The systems paradigm views software as interacting components forming a whole; real stuff doing real things. The language paradigm views software as abstract signs and rules of grammar conveying meaning. Good design, from a systems perspective, comes from a skilled technician following good design principles (I would even call it aesthetics). Good design, from the language perspective, comes from a relatively less skilled technician working within a language that from the outset excludes bad design through grammatical rules and compilers. The system approach tends to view software as a living organism that is incrementally poked and prodded, changed and observed. The language approach tends to view software as a series of mathematical transformations, preserving meaning. If each of the paradigms were a theory of truth, the systems paradigm would be correspondence, and the language paradigm would be coherence.

I see system versus language as engineering versus science. I view engineering as a bottom up, incremental, tinkering approach, at least when it comes to software and the way I like to practice software engineering. I view science as a top down, formal, mathematical approach. I actually like both, and I think both have their place, but when engineering is made subservient to science, we're actually losing something very important. When engineers are shut out of conferences and journals, there are discoveries that will be left unpublished, and new scientific theories left untheorized. (This was what Gabriel saw happening.)

Computer Programming as an Art

"Computer Programming as an Art" by Donald Knuth
http://dl.acm.org/ft_gateway.cfm?id=1283929&type=pdf

For those with even a cursory exposure to Computer Science, Donald Knuth needs no introduction. Knuth is coming from an academic perspective, but even for an academic his perspective is a bit unique. He has created and maintains several large open source software projects. This is his ACM Turing Award lecture given in 1974. He starts by quoting the first issue of the Communications of the ACM (1959). It claims that for programming to become an important part of computer research and development (to be taken seriously) it needs to transition from being an art to a disciplined science.

The big idea I draw here is: Programming can be art (in the "fine art" sense), which means it is (at least sometimes) a creative endeavor.

Knuth first explores the definition of "art" and "science." He looks at their use over time. Their use was (and is) not consistent. At times "science" and "art" are used interchangeably. "Art" was used to describe something made of human intellect, not nature. Eventually "science" came to mean "knowledge" and "art" came to mean "application." Though even that usage is not universal. To Knuth an "art" is something that is not fully understood and requires some aesthetics and intuition. A "science" is something well understood. Something that can be mechanized and automated. It is something that can be taught to a computer. Can computer programming be taught to a computer?

Knuth does not think that programming can ever be fully automated. However, it is still useful to automate as much as possible, since it advances the artistry of programming. He believes, and cites others, that progress is made not by rejecting art in the name of science, nor science in the name of art, but by making use of both. He makes reference to C. P. Snow's "The Two Cultures" as an example of another voicing concern about separating art and science. At this point when he speaks of art he means something more along the lines of "fine art" than "engineering."

Knuth goes on to talk of creativity, beauty, art, and style. He hits on how sometimes resource constraints can force a programmer to come up with an elegant solution, and this has an artistic aspect to it. He also encourages people to, when it comes to programming, make art for art's sake. Programs can be just for fun.

Knuth's talk is focused on the act of programming, and when he deals with engineering versus science he means with respect to the act of programming. To what extent can the act of programming be made automatic? To what extent must it remain a human act of creativity? This is a little further afield of the previous sources, but Knuth's insistence on seeing programming as a creative act is the big idea I drew from his talk, and is really the point of his talk.

Given that programming can sometimes be a creative act, it raises a lot of questions in my mind. Is programming always a creative act? If programming is a creative act, how should a programming project be managed? Is the high failure rate of software projects related to this? Perhaps this ties back into Tom Simpson's "Masterpiece Engineering" satire. Imagine a project manager with a room full of artists creating Gantt charts and task dependency graphs to plan out the creation of a new masterpiece!

On the other hand, nothing appeals to the ego more than seeing oneself as a grand master of art. There should be a measure of moderation here. I think there is benefit to trying to understand programming as an artistic (or at least "creative") endeavor, whatever that means, but we should not go crazy with hubris.

Better Science Through Art

"Better Science Through Art" by Richard P. Gabriel and Kevin J. Sullivan
https://www.dreamsongs.com/Files/BetterScienceThroughArt.pdf

"Better Science Through Art" by Richard P. Gabriel
https://www.tele-task.de/archive/video/flash/12636/

I have already covered some of Gabriel's background, but I will say that having been involved and educated in both a technical field and an artistic field gives him a unique perspective on the relationship between science, engineering, and art.

I unfortunately don't know much about Sullivan's background, other than he is a professor of computer science at the University of Virginia. His collaboration with Gabriel produced one of my favorite papers ever. I don't know that I can tease out what should be attributed to whom. I will be basing my comments on Gabriel's talk, but I don't intend to attribute everything to him, or to diminish Sullivan's contributions.

The big ideas I drew from this is:

1. Science, engineering, and art all have at their core "disciplined noticing."
2. Disciplined noticing is a skill that requires practice.
3. The creation of knowledge—even in the case of science—requires an abductive leap powered by creative spark.

This is a really great talk, and covers a lot of ground. It is entertaining, insightful, and very worth watching. He attacks some common caricatures of science, engineering, and art, and digs into the actual process behind each. In the end, he finds that there are a lot of similarities to the methods in science, engineering, and art. It is a process of exploration, discovery, and verification. He calls it disciplined noticing.

I have found this to be true in my experience. Just like people have a caricature of science, that it is straight line progress, the monotonic aggregation of knowledge, there's a similar caricature of software development. My experience has been that writing software is a creative, exploratory process. Sometimes I go down an alley, but find that I need to back out and take a different turn. I may write a test, run it, change some code, change a test, run it, think for a while, delete a bunch of code and rewrite it all.

In my experience this process—writing, evaluating, and rewriting—has much more in common with writing a novel than constructing a building.

Conclusion

This long meandering post must come to an end. First of all, I would highly recommend looking at each of these cited sources. They will reward you. Perhaps you may even find that I have seen them through my own preconceived notions, and you may draw an altogether different conclusion from them. So be it.

This "conclusion" is not really a conclusion, but a way-point. I started on this journey to understand the nature of software engineering, how it is different from other kinds of engineering, and why it is so hard. I ended up at a place that intuitively I knew I would end. I will not make an absolute statement. I will say that at least sometimes (and in my experience) software development is a creative process more akin to creative writing.

I have also seen that there is a tremendous amount of creativity in both engineering and science. I believe that at the core of engineering, science, and art is a drive to understand and influence the world, which requires observation, testing, and evaluation. I don't claim to know how to do software engineering "right," but I don't think we will ever do it right if we refuse to see that creativity (which is at times unpredictable) is a key part of the effort.

I have learned that both engineering and science are useful for discovering and validating knowledge. Scientists and engineers should collaborate. Neither should be seen a primary at the expense of the other. They can even be seen as external expressions of the same process sometimes using similar tools and techniques.

I have learned that software is unique in engineering. Whereas a blueprint is a written artifact using specialized notation, the building it describes must be brought into existence through a complex error prone process. Code is written using specialized notation, but the gap from code to execution is much smaller. There are pitfalls and challenges, no doubt, but I would like to see how the nature of what we produce can change how we produce it. I'm still holding out hope that the nature of software can change the face of the human organizations that produce it.

Practically, what this all means is that a software engineering process should be iterative. It should embrace unpredictability and allow space for the creative process. In the same way that a painter never thinks his painting is complete, software should be developed in a way that continuously produces value, so the project could be closed down and the product shipped at any point, and the customer is still happy with the result.

So I end back at the beginning with Vanderberg. I don't think that agile is the last word, but I think it is the best we have so far.