The Long Road Home

I’ve toyed with a few titles for this post, including - ‘My Kingdom for a Backslash’, or ‘Fish Don’t Know They’re in Water’. In the end I felt ‘The Long Road /home’ was the most fitting. This post is about my thoughts and experiences as a software developer, leading up to my use of Linux and building my first non-Microsoft based Web application.

In The Beginning

I’m a non-CompSci grad, having started my academic life with a college diploma in biology (a Canadian community college is the equivalent to what would have been called a ‘polytechnic’ in the UK). After about four years of working as a laboratory research assistant I decided that biology wasn’t for me and moved into IT. I started the hard way – PC support, and slowly worked my way up from there. I had a couple of lucky breaks along the way, which combined with a lot of work meant that I was eventually able to pass for a nearly competent IT professional.

After several years of ICT related employment and having overseen a couple of medium sized software projects, I decided that I needed to go back to the coalface and learn more about software development. I started by writing desktop applications in VB6, followed by C#  and ASP.NET . I also sat the required exams for the Microsoft MCSD and MCDBA certifications (.NET versions). I was firmly in the Microsoft camp. In fact even before my switch from biology to IT I’d been building XT compatible clones – feeding DOS and Windows floppies into the disk drives of my home-built machines. It all seemed like good fun at the time.

Later, having earned some certs, and with a few projects under my belt I thought I was doing all right, earning okay money on a couple of contracts and thinking things were generally headed in the right direction.

Ch-Ch-Ch-Ch-Changes

But then two things happened. The first was that Microsoft unleashed a wave of frameworks and technology – part of what was then called WinFX including Longhorn, WPF (Avalon), WCF (Indigo), WWF (Workflow). The second was that a few years later I decided to go back to school. I remember thinking at the time that by going back to school I was simply going to rack up a decent qualification – something that would help me to progress and maybe even help prepare for the day when I would hang up my keyboard and move back into management, or something else altogether.

What I hadn’t anticipated was how hard it was going to be to study as well as stay on top of what was happening in the Microsoft world. Things were changing fast, and I was struggling to keep up. Like some of my contemporaries I was also trying to stay abreast of the broader trends in software development. I’d started reading some of Fowler’s  stuff. And I was reading a lot about the challenges of managing software projects – challenges that are unique to building systems that reach out and touch all aspects of an organization. I’d read ‘GoF ’, Kent Beck , Eric Evans  and others. I watched the TDD , BDD , and Agile  zealots rise to glory. I was also following the raging WS-* debate, and beginning to understand the significance of Roy Fielding’s  work.

I was beginning to acquire a deeper understanding of the software development process as a whole - in particular an appreciation that the major challenges for most projects are not technology related. To this day the two most important texts I’ve read on the topic are The Mythical Man Month  by Frederick P. Books, and PeopleWare  by Tom DeMarco. Incredible when you think that The Mythical Man Month was first published in 1975. Read them both and you’ll understand why they’re important in the field of software and systems.

Somewhere between studying and trying to keep up with things in general, I began to experiment with Linux. I’d just started a course on computer security which included Linux and Windows. Thanks to virtualization technology, spinning up a desktop Linux VM was a breeze, although apart from using the VMs for a couple of exercises on my course not at lot else happened. I was still a bigoted Microsoft developer, convinced that C# was a real developer’s language and that Linux, Apache, MySQL and PHP were for kids. Actually that’s a little harsh. I wasn’t really bigoted, at least not in the literal sense – but humans are a tribal bunch, and we like to stick to what we know. Often when our beliefs in what we know are challenged we resort to derision and are dismissive of alternative points of view.

Things began to change when my computer security course started looking at historical operating systems combined with some basic principles of machine organization and computing. It was then that I realized my entire frame of reference in computing had been shaped by the tools, messages, and strategy coming from Redmond. It was a myopic point of view. I discovered that a lot of the messages coming out of Microsoft were simply rehashed versions that had originated elsewhere and were designed to keep people on its platform. When Microsoft decided it was time to push a point of view that suited its technology, it did. And as far as message delivery goes – it excels at it. Remember the 'impedance mismatch' between general purpose programming languages and SQL? That was the message de jour when LINQ was released. Why not earlier? It was weird to see so many Microsofties ‘on message’ at the time. Of course Redmond wasn’t the only thing influencing my cognitive map of software development. My up-to-date view of what was going on in the development community was now largely shaped by the long list of RSS blog feeds I’d subscribed to. I learned a veritable ton from excellent bloggers and technical writers.

Another important event in my transition from a Microsoft-only approach was when I attended my first Barcamp  developers' conference here in Bangkok. I was impressed with the energy and enthusiasm shown by the small army of open source software developers I’d met there. I wasn’t quite ready to put on my Richard Stallman  acolyte’s robe (it turns out not many are, but that’s a different story and the work that Richard Stallman has done is hugely important), but there was some very cool stuff going on. I also couldn’t help but notice that many of the successful Microsoft open source projects were in fact ports of successful projects that had come from other languages and platforms; like Lucene, Hibernate, and Velocity to name a few.

The tipping point came when I was involved in a large .NET project that can only be described as the quintessential death march. Death march projects aren’t unique to users of Microsoft technology, but this one happened to be using those tools. No one in the organization had heard of Richard Stallman , or REST , or had even the slightest interest in Agile; not that those alone are the basis for establishing the maturity of a software development team, but it was an indication that the team – like a lot of other teams, had been living in the Microsoft echo chamber for too long. For me personally – it was hard – really hard. And most of what was hard had nothing to do with writing code. There were problems in requirements, which had led to classical problems of oversight and realizations – the stuff of any difficult project. But what I found interesting in hindsight was the level of friction in tools, frameworks, testing and deployment – all of which conspired to fight against each other creating an exhausting drag on the project.

Lessons Learned

After a bruising project it was time to complete my studies and write my thesis. I’d sworn privately that I would never allow myself to become worn down or as affected again – even if it meant a fundamental change in direction (although not quite ready for 410 Gone ).  Fortunately, my next role was a fairly low-key part-time engagement which allowed me to devote the rest of my time to writing my final report.

What transpired over the course of those happy nine months was that I built and deployed a Linux-based production server and wrote my first Web application based on an open source framework.

And after all of that, this is what I’ve learned:

First, Linux , Apache , PostgreSQL  and PHP  are not (just) for kids. Linux has turned out to be the lion of the Internet. Look at the latest Netcraft Webserver survey  - 65% of the Web is powered by the Apache HTTP Server . Back in the late 90s and for most of the 2000s (and while Microsoft was still mucking around with IIS4, 5 and 6), the Apache HTTP Server community was benefiting from a world class webserver along with all of the other tools and technology that fit together to form seamless and friction-less development and deployment environments - all part of the Unix philosophy, and long before Microsoft developers even began talking about ‘friction’ in their projects.

I’ve come to realize that what’s happening now in the *nix world is simply too big to ignore and I wouldn't work for, or consider hiring anyone that didn’t possess at least a rudimentary understating of this fact.

Second, I mentioned above that the major challenges for most projects are not technology related. In one of the first texts I’d read by Fowler  he describes three fundamental risks to any project: 1) requirements risk, 2) skills risk, and 3) technology risk. I think requirements and skills risks conspire to create the most trouble for any project (although technology and associated risk is still undeniably important).

Successful projects can be, and are run using Microsoft technology, but remember this: Microsoft’s only objective is to sell software licenses, and to do what ever is required to continue to sell software licenses. And that is all. It’s a company that has arguably been over compensated for decades from a license-based revenue stream that is the result of its dominance in the desktop computer market. Nor has Microsoft been shy about using its position of dominance to destroy competition, and in the process innovation. Its attempt to ‘create’ a developer community is a top-down effort and is the antithesis of how communities should be formed.

The developer division in Redmond still largely decides strategy for the development of Microsoft-based frameworks and technology. When compared to the level of activity and innovation in other communities it might be reasonable to describe a dependency on the Microsoft platform as a technological risk – both in terms of vendor lock in, and in terms of the maturity of the development community as a whole, and that these risks should be assessed as with any other risk to a project. If a project is at a stage where it can freely choose its underlying technology, then I believe there are healthier tools and communities in the non-Microsoft world.

Two great cases in point are the Ruby  and Node.js  communities on GitHub  (and elsewhere). Looking at the speed of development and contributions to either of these communities, and it’s clear that while feathers might still be ruffled – good projects and tools quickly rise to the top, while mediocre projects fall to the side. It’s fascinating to watch.

Third, anecdotal evidence suggests that as many as 70% of all software projects fail to meet their stated objectives. It’s not a fun industry to work in, and death-march projects will continue to exist in all of the major technological camps. The success of any project will be determined by a subtle combination of maturity, experience and insight along with realistic estimates of the true cost of creating or implementing software-based solutions. In theory, the up-front analysis of a project is meant to be technology agnostic. In practice, most shops are in one technological camp or another. I think at this point however, at least one indicator of the likely maturity and experience of a potential supplier, or team – is their genuine willingness to consider a range of technologies and solutions including open source platforms and tools like Linux , PostgreSQL , PHP , Python  and Ruby  to name a few.

Fourth, I wish I’d started my career in software development at the 'Web face', maybe as a Perl/CGI scripter.

Over the coming months I’ll be publishing a series of blog posts on how I got started with Linux , Vim , and a bunch of other tools that for me at least, made computing fun again. Stay tuned…