Making Order from Chaos

One thing I have noticed throughout my career is the increasing absence of “systems analysts”. In the old days, there used to be folks, separate from the programmers, whose job (at least in theory) was to have a foot in both the technical and the business camps. The idea was that they could speak the language of both side.

Analysts were there for a reason. Software development requires things to be defined in a pretty orderly way, whereas the business world is a pretty chaotic place – it doesn't have all the clearly defined rules and structures that programmers wish it had. The role of the analyst was to try to make order from chaos.

The analyst could speak to the business folks in their own terms, elicitiing just enough formality to work out what was needed, and at the same time speak to the technical folks with the precision and certainty they hope for.

Analysts have gone out of style. Nowadays we have the generic term "developer": a technical guru who can also speak directly to customers. Perhaps the idea of a “do it all” developer came about when object oriented languages made us think we are modeling the world as it really is.

Unfortunately, in the business world the objects in the real world are often not easy to work out unless you know a lot about the business area you are working in. Therefore, you are supposed to ask the business experts. Unfortunately, the business experts are often experts at doing their jobs, but often don't have the necessary experience or inclination to "pin down the rules" the way we would like.

The answer is, of course, for the developers and the business experts to be able to work together. This is where we find that most developers sorely lack the “requirements gathering” skills that good analysts developed over many years. Developers tend not to be good at dealing with "fluffy thinkers" and race to capture the “domain objects” as quickly as possible. To achieve this, many developers grasped at graphical object oriented notations to “express” requirements agreed with business experts.

Getting the business folks and developers talking is a good thing. But focusing those discussions on capturing requirements in terms of unambiguous object oriented diagrams meant that developers often ended up viewing the diagrams as something that can be translated straight from boxes and lines into code. Supporting this view, a number of software tools vendors advocated the effectiveness of “round-trip-engineering” wherein the diagrams and program code are simply seen different notations for the same precise model.

The goal, then, became getting the diagrams tied down, so the developer could then directly reflect them in program code.

This meant we didn't have much analysis going on any more – rather, the focus was on demanding a diagrammatic specification, adorned with supportive text where needed, that was precise enough to be translated into a programming language.

This led inevitably to big fat object-oriented diagram-heavy specification documents that most business folk could never write well, and that most developers often complained had big holes and contradictions. This meant that despite the “progress” neither side was happy.

Agile methodologies came to the rescue with the obvious idea that big problems can be solved in small steps - focus on a bit a time and your chances of going wrong are minimized. Furthermore, if you keep the customer closely involved in each step, and measure your progress to make sure everybody is happy, people can see good progress, and even learn from each step as a way of decided what the next step should be about.