Friday, December 29, 2006

Standoff

I felt that from now on I needed to get closer to actual customers, to find out what their real needs were. My idea was that if we understand what the actual customer wants, delivering real business value in their eyes, then we would have a much better chance of delivering software useful to the business.

Consequently, I moved on to a successful Silicon Valley consultancy, focusing on financial systems far large (and rich) banks. They suggested a variety of projects, and I jumped on one that gave me daily exposure to real banking experts.

We were to produce a replacement for a legacy banking settlement system – which consolidated banking trades and redistributed money and ownership of stocks and so on between banks. It is a very profitable business, and our client had a large budget for the development of the replacement settlement system.

After being offered the job, but just before starting it, I took a “road trip” through California, to unwind, and prepare myself for my imminent escape from messy projects and (at last) my exposure to the “real way software is done” in businesses that know what they are doing.

My hopes were quickly shattered. It turned out that the project had been going on for a year or so when I joined, and it was stagnating at the uncomfortable stage where the bankers kept asking the developers “when are you going to deliver the software?” and the developers kept asking the bankers “when are you going to give us the requirements?”

This standoff soured the relationship between the bankers and the developers into an “us and them” mentality. I noticed that the developers tended to keep busy with technical distractions, and the bankers tended to keep busy by avoiding us. Still, the bank kept paying our high daily rates and per diems, and the general approach seemed to be to keep ours head downs and not make too much of a fuss.

I was there for two years, and the standoff hardly improved throughout the whole of that period. Every now and then, of course, we would produce some deliverable, mostly for appearances sake. Most presentations were cast in technical terms that could rarely be tied back to any real business need. Still, what did they expect when even the business experts couldn't tell us the requirements?

Eventually, the big bosses in the bank picked up on what was going on, and the project was scaled back dramatically, and (I heard) eventually scrapped. Of course, we continued to blame the bankers for never giving us any requirements, and they blamed us to the very end for never delivering anything of real value.

I suppose that in the big scheme of things it didn't matter: the consultancy had received a nice profit from the bank, and the bank continued to prosper despite the project. Personally, though, I was very feeling pretty low.

Smart Suits

After a little hunting around, I became “world wide chief scientist” for a pretty large software development organization, with offices around the world. Officially, I was responsible for all software products. In practice, I was only responsible for the exciting “next generation” ones. This gave me another great technical refuge, where instead of speaking to real clients I spent most of my time surrounded by senior technical people from our various offices forming a mutual admiration society.

To be honest, this was a quite different kind of company from my previous employers, since the chief executive himself was deeply involved in ensuring developers had “clear focus” throughout their work. He made sure we had a unambiguous and unwavering requirements up front, then gave us four years to deliver a system matching them. During that four years we only had one business meeting involving anybody outside the development team, and that was to discuss salary reviews. As far as we were concerned, requirements were fixed, and now we could remain focused on real software development without any outside interference or moving goalposts (as we called unwanted requirements changes).

We had no uncertainties. We “got on with it” and delivered on exactly what he wanted. The chief executive was delighted. Here, we felt, was a “smart suit”. He knew what he wanted, expressed it clearly, and provided an isolated environment where we could deliver on our promise. He boasted at conferences about how great our development process was, and what a great job my team had done. We were heroes.

The downfall came a couple of years later, when it slowly dawned on that that although we had faithfully delivered precisely the product the chief executive has asked for, hardly any customers actually bought it. We produced something without mess or uncertainty, and nobody wanted it.

We had assumed that development was about pinning down requirements up front, and delivering whatever those requirements dictated would lead to success. We hadn't worried about whether or not those requirements were relevant to the target market. We has assumed that “if you build it, they will come”. Well, they stayed away.

My painful lesson from this experience was that a “pinned down” requirements specification doesn't make your worries go away.

Dumb Suits

After about a year at the military contractor, I took a day off work to attend a computer conference to see if other companies worked the same messy way. There I found lots of CASE tools and tidy Methodology Manuals showing the “right way” to do thing. Strangely, one of the presenters was from the very company I was working for, with one of my colleague presenting the CASE tools and reliable methodology that “we use for all our projects”. Maybe I had been on unusual projects, but none of the dozens of people I was working with had ever used any of the tools of methodological steps we were presenting. I began to suspect that the same was true for many of the other tools and methodology vendors. I began to wonder if most projects had a much messier reality than their public face.

At this conference, I started chatting with a manager for a large American telecommunications company. It turned out we both had a common acquaintance, who recommended that the American company employ me, and a month later I was (joy of joys) in New York working in their research labs.

What bliss this research was: I could focus on technology without any worries about the dirty messy world of non-technical requirements

I had colleague who spent years working on speech recognition systems, and neural networks, just because these things seemed sexy. I had others working who tried to be at least “grounded” in their research and focused on virtual memory management performance optimization because it seemed potentially useful in some real world setting.

My own work, by the way, drifted accidentally into early research on high performance object oriented programming languages – simply because the dazzlingly intelligent guy heading up this group had a vacancy for which I was the only applicant. We worked for years coming up with all sorts of novel techniques for compilers, language runtime environments, hybrid programming languages (Lisp merged with Prolog, Smalltalk merged with C, etc.) and thought we were conquering the world.

As much as I enjoyed this indulgent lifestyle, I was constantly worried that the research was benefiting the intellectual curiosity of us researchers, but not much of it was really benefiting anybody else. My colleague reassured me that big businesses always had big research budgets, and most never expected most of it to come to anything useful, the underlying hope being that one day some of the geniuses would come up with something earth shatteringly great.

Not much later, the big bosses from head office looked at the research projects in our labs and said “clever, but irrelevant”. They shut down our whole division, putting hundreds of people out of work.

At first I was astounded at how short sighted these guys were; how could they not understand the staggering impact of our technical breakthroughs? Deep down, of course, I knew that all along we had been working on things that pleased us, but never gave much thought to what really mattered to business. Publicly, I blamed the “suits” for never telling us what they wanted. How could anybody produce anything relevant to the business, if the “suits” never gave us any requirements and just let us wander on our own technical whims? At the time, I was sure that this only proved that we were smart, and they were dumb. I decided to move to somewhere where even the suits were smart.

Thursday, December 28, 2006

Get On With It

When I first started out doing software development, I worked for a military contractor developing an aircraft parts warehousing system for the Royal Air Force. My role was quite junior: I was given database schemas created by the older and wiser analysts and my job was to create “input screens” for maintaining the data in the database.

The analysts left as soon as the schemas had been handed over to me, leaving me alone “on site” to create the screens. I had no idea what people in the Airforce actually wanted the system to do, but maybe that wasn't my concern. Still, it worried me a bit, so I phoned my boss back at head office to find out. He told me to not worry about it at this stage and “get on with it” for a month, after which a senior analyst would visit me to review my work.

For the next couple of days I wondered if I wasn't smart enough for the job: I knew I had to “get on with it” but I just didn't know how. In the end, I did the only thing I could think of: I distracted myself by playing with technical features of the database until I at least knew something useful to me (if not actually useful to the eventual system user).

After a while digging deep into the technology, I found that I could create “input screens” really quickly, and even invented a “framework” that made it simpler still. The framework interrogated the database schema and generated most of the necessary input screens automatically, which I then only had to “hand tweak” a little. Focusing on technology proved to be a satisfying refuge for me – allowing me to keep busy and have fun at the same time, and push aside some of my nagging doubts about the usefulness of my work.

Within two weeks I had dozens of input screens. Within a month I had them all. Of course, I was still a little worried that I was alone doing all of this, and nobody had given me any feedback about what the customer really wanted, but at least I was making progress. I would save up all my doubts for the senior analyst's visit.

Right on time, the senior analyst arrived to “check on my progress”. The first thing he did was take me to lunch. It was a nice meal. I asked him lots of questions about my worries, and he promised he would look into that when we got back to the office. Back at the office we sat down at the computer and he asked what I had been working on. I told him, and gave him a short demonstration of the input screens. I also asked him a stream of questions along the way, and he admitted he couldn't answer them. I was disappointed to find out that out he didn't know anything about the project at all. His job wasn't to help me out, but to check that I had at least been doing something during the past month. At the end of the day, he told me how impressed he was. Before he left, I asked him about getting some feedback on the actual requirements, and he told me I was scheduled to give a demonstration to the client in a week and could ask my questions then.

A week later, I sat in a conference room with lots of senior managers from the military contractor. They told me I was going to show the system to the most senior officer on the airbase, and I shouldn't make waves when he was around because we were “down to the last two” in the shortlist for getting this project and this project was “worth a lot of money to us”.

The senior officer arrived, and I gave my demonstration. He smiled a lot, and shared jokes with my bosses. We all went off to lunch, and they swapped stories about people I had never heard of. After lunch the airforce officer asked me to demonstrate the system to his junior staff who would actually be using it to manage the storage and movement of aircraft parts around the world. Well, at least I now knew something about the requirements, if only a tiny morsel.

The junior staff arrived, and within five minutes pointed out that the system was nothing like what they had hoped for, and was far from what they needed. They told us that it appeared even less effective than the decades old manual system we were aiming to replace (which was news to me).

My bosses squirmed in their seats, and cleverly suggested we “complete the demonstration now and return to the detailed design discussions another time”. We finished the demo, the junior staff left, and the senior officer told me and my bosses “I will take care of my staff”. Which my bosses later translated for me as “he will keep the trouble causers quiet”. He did, and we never had any more complaints about the system not being what they wanted.

I continued to work on the system for a few more months, and then moved on to another project (an even more frightening one managing fissile materials). Another team took over where I left off on the aircraft parts warehousing system, and took the screens I had developed, “finished them”, and delivered them in exchange for a very large fee.

I don't know what happened with the system in the long run, or how on earth the junior aircraft staff managed to use it. I did hear, however, that they were still using the old manual system as a “backup”.

The whole experience left me feeling very uncomfortable. I thought it was a one off, and then spent the next twenty years finding out that it wasn't.