penitent, turbulent, multiplex!

In day-to-day conversation with other developers, a lot of points of debate tend to surface fairly often. A handful that I’ve run into recently:

1. XML should/shouldn’t be used as a data storage format
2. The best/most maintainable way to approach concurrency for performance is X
3. The best/most maintainable way to approach concurrency for responsiveness is X
4. Object-oriented programming is better than / worse than / the same thing as functional/straightline programming

And so on.

One of the few things that graduate school taught me is that a lot of problems that identify as “new” have actually been treated rather thoroughly somewhere back in the annals of time. It’s somewhat embarassing to have to learn this lesson as a software engineer, I think, since we’re a pretty young field as far as intellectual disciplines go.

I suspect that our ignorance of the past is because a lot of the good discussions surrounding them are in the primary literature, which for some reason tends to bring out a stubborn streak in many of the practitioners I know. I think that by suggesting articles from academic journals, some might feel that I’m being snooty, stuck up, or otherwise just showing off my education. I haven’t quite figured that one out yet. But it happens.

However, I feel like people are missing out. There’s some really good stuff out there. For example:

• To address point 1, the aptly-named Michael Stonebraker wrote a great paper on why XML is a terrible data storage format. (Thanks to Rachel Pottinger, my data management professor and supervisor.)
• When researching point 2, I came across a bunch of wild algorithms from the early 80s for parallelizing problems that appear inherently sequential, like searching linked lists. These were written for the Lisp machines, which had hundreds of 1-bit “processors”.
• When talking about point 3 in an operating systems course, we read a pair of papers (here and here)that each put forth reasons why events or threads were more appropriate for this task.
• Point 4 has been discussed cogently in some very old books, many of which I babble about incessantly (the Structure and Interpretation of Computer Programs, and Object-Oriented Software Construction).

It seems counterintuitive to me that so many current problems could have been so well addressed such a long time ago, but I guess it’s the nature of academia to predict (or invent) problems long before they actually become a pain point. The result of this is summed up nicely by Michael Stonebraker in the paper I linked above:

Most current researchers were not around for many of the previous eras, and have limited (if any) understanding of what was previously learned. There is an old adage that he who does not understand history is condemned to repeat it. By presenting ‘ancient history’, we hope to allow future researchers to avoid replaying history.

I think this one goes without saying in almost any discipline. Having a mentor — someone who is wiser than you in some aspect of what you are doing, and hearing out their suggestions — will always stretch your brain. Of course, it can take a while to find someone you click with and who also has the time for you.

While you’re hunting for a mentor, you can get the next-best thing by reading the writings and the code of “famous” software engineers. Books like Programmers at Work and Beautiful Code are great for this sort of thing. And tons of very clever people write open-source software that you can peruse.

While seeking a mentor is a somewhat obvious way to improve, inverting that relationship by looking for a mentee — even when you’re very green yourself — is, in my opinion, a great way to improve as well. Einstein’s old adage of “You do not understand something unless you can explain it to a six-year-old” comes into play here.

I’ve had quite a few opportunities to teach intro-level computer science and software engineering courses, and these experiences clarified a whole bunch of things for me in a hurry. As I sought explanations and examples that would click well with an entire class, I started to learn what methods of organizing programs in the small would communicate well across a broad audience. I try to carry those lessons over into “real-world” projects, so that other developers can (hopefully) make the appropriate modifications to work I’ve done regardless of their skill level.

Plus, working with people who are just starting their journey is refreshing. Their optimism and the breadth of (often strange) approaches that they take to problems that to us are old hat is a lot of fun.

So, get out there! Give, take, interact. Humans are the interesting part of this craft, after all.

This should go without saying, but it’s surprising how long I went without actually doing it.

A few years ago, I read Code Complete, which I’ve talked about at length here before. One of the first things that impressed me about that book was that it laid out a very precise reading plan that all newcomers to McConnell’ company had to follow.

Until that point, I’d mostly picked up new books and tried new techniques whenever I felt like it. If I heard of some reference or thought of some activity that would be beneficial to my learning (e.g. reading book X, or doing programming exercise Y) I would jot it down in a looong to-do list, and hope to get to it eventually.

After I saw McConnell’s suggested reading list, I went back to my own bucket of learning items and loosely organized them into a sequence. I also defined what it would take to actually get the desired benefit out of each activity. For books, this meant anything from skimming the text to doing all of the exercises contained therein. For extended exercises or activities, I gave a bit of thought to the scope that would be required to actually learn anything from the attempt.

This has ultimately resulted in me getting more out of the time I spend learning about my craft. For example, I’d previously read the Structure and Interpretation of Computer Programs. It only took me a week, but I’d be hard-pressed to tell you anything about the last four chapters of the book.

I then read it again, about a year later. This time it took me significantly longer to finish, but I hadn’t retained much of it. I decided that, in order for this activity to be worthwhile, I’d have to do a significant portion of the exercises in the book.

So now I’ve been working on SICP for about a year, and I’m not even halfway done the work I believe I will have to do in order to consider myself done with it. Yes, that’s a very long time. But, the value I get from each session is much higher than it would be if I was just skimming the same material. That is, every time I sit down with SICP and a stack of paper, I come away an hour or two later feeling a whole lot wiser than when I sat down.

I feel that the few hours that it took me to organize my list of “things I’d like to learn about” into a loosely annotated and ordered plan has helped me a lot. Hopefully it might help you as well.

When your average software developer goes to break a subsystem or module into implementable pieces, generally her first instinct is to think “What are the classes and behaviours in my problem domain?” Whether you’re driving this design with TDD, or taking a more top-down approach, this question tends to be the linchpin of the design.

This is unsurprising. Object-orientation is (ostensibly) taught to most of us in school as the one-true-way of doing things. It’s rare to find people who do not equivocate software-design-in-the-small to object-oriented design.

A couple of years ago, I decided that I wanted to learn “functional programming”. My first serious attempt was in OCaml. However, once the wheels hit the ground, I felt like I had returned to highschool. At that time, I knew the syntax and vocabulary surrounding objects, but I had no idea how I was supposed to use them.

So, I reasoned, I needed to learn how to design functional programs. I needed to learn “functional design”, for lack of a better term.

After a bit of consideration, I remembered that there are a lot of very famous universities that teach the basics of program design using Scheme. (This is also true for OCaml, but the Scheme ones are arguably more famous .) This was perfect — I needed to learn how to design programs all over again, in a “functional style”. So, why not go back to basics? And thus did I seek out MIT’s CS1 old-style curriculum and the Wizard Book — also known as The Structure and Interpretation of Computer Programs (SICP). (They’ve since switched to Python, I think. I imagine the spirit is the same.)

You may notice that I’ve been putting “functional design” and “functional style” in quotations. The reason for this is that I quickly discovered that such things do not actually exist.

Let me explain.

The fundamental exploration that jumped out at me in SICP was this: Let’s say you have a programming model that supports first-class functions, and does a pretty good job of supporting immutable data. What reasonable methods of designing and implementing programs can we arrive at using this model?

And the answers are legion. A program can be viewed as a mathematical function. It can be viewed as a signal-processing device over finite sequences of discrete signal elements. It can be viewed as a signal-processing device over infinite sequences of discrete signal elements. It can be viewed as a processor of a domain-specific language. And so on.

What SICP really shows you is a variety of methods that you can use to design programs-in-the-small, with object orientation being only one of them. So now, when I’m working on a new chunk of some big project, I find myself thinking not “where are the objects”, but instead “what happens if I view this problem as, say, a signal processing problem?” Even more importantly, when I read someone else’s code, I can ask myself the same thing.

The reflex of explicitly asking myself this question when faced with a new design problem really helps to understand it from multiple angles, and to identify the things that are going to be the hardest to model or extend in any feasible design. And that lets me focus on clearing up the things that matter.

I won’t pretend to be a master of any design-in-the-small methodology. However, I’m now aware of a whole set of possibilities that I was once largely ignorant of, and this knowledge helps me in a bunch of little ways every day.

In the last short, I used several analogies. I likened the craft of a software developer to that of a contract artist, or a shoemaker.

Analogies are useful things. When I have the pleasure of sitting down with a group of people who are really excited about what they do, and who are discussing a very challenging problem, what happens almost immediately is that someone will draw an analogy to help the discussion move forward.

This definitely gets the conversation moving. It’s the direction that is the rub. It’s really easy to let an analogy run wild, to the point that it gives you almost no insight on the problem you were initially trying to understand.

As an example: Just the other day, I was at a great lunch meeting where the discussion was centered around software testing. Someone mentioned that developers often write software with the assumption that the user is usually going to do things in a very predictable fashion, and that ‘weird’ cases are exactly that — weird. In this sense, the developer is living with the same set of assumptions as a traditional performance artist, where interactions are scripted, and mess-ups are assumed to be few and far-between.

It was mentioned that perhaps a better mindset would be that of the improv performer, where the fundamental assumption is that everything is going to be unexpected. The analogy was a good one — it clarified that there were two sets of axioms we could be working against, and perhaps that one of them was more appropriate than the other.

Within twenty minutes, we were talking about this one time that someone had been to a play where the lights had gone out, and the crew decided finish the performance by candlelight. The analogy had run amok. We were certainly still having a stimulating conversation, and possibly there was a lot of value in what was being said. But we’d completely lost the original point.

Usually when I’m involved in a conversation that is centered around an analogy, I try to bind the concepts within it to concepts in the topic that we’re actually trying to understand. When I hear a lot of those threads breaking, that’s how I know the analogy is taking on a life of its own.

Why do I think this is so important to being a good developer? I have two reasons. The first: When I encounter software that I think is particularly pleasant to work with, it is because the high-level metaphors that are being used to describe things are obvious and are used consistently throughout the product. If you look at a lot of patterns in software design, they’re essentially dressed-up analogies. (A “bridge” is sort of like a physical bridge because etc… an adapter is sort of like a power adapter because etc…)

The second is that I’ve observed that many smart people that I like to learn from tend to use analogies heavily. I don’t often get a lot of time with those people. And so, to really make sure that I’m getting the most out of my time with them, I like to carefully monitor the evolution of their analogies. After all, I can spend my afternoon talking about botched performances with pretty much anyone, really

(Steve McConnell does a good job of dissecting several analogies used to describe software development as a whole in his massive Code Complete, 2nd edition. I can’t say that I particularly like the one he finally arrives at (“it’s like construction”), but at least he is thorough in itemizing the components of the analogy that work, and the ones that don’t quite mesh well. It’s that sort of exploration that really helps you understand the original problem, in my opinion.)