I’ve been re-reading Venkatesh Rao’s writings on the Gervais principle. It’s a depressing and disturbing read. What I’m trying to do at the moment is reconcile it to the picture that I’ve got of how software development is done inside financial services companies – my own recent experience in the world of work.
The picture I’m going to talk about here is an amalgam from several financial services companies, retail banks, insurers and credit card companies. And it really isn’t worth singling out any one of them as being particularly bad, because they are all very similar and - as I’m about to theorise, it may be that they could simply not be any other way.
The idea that software development in a particular financial services company was as bad as it possibly could be drifted into my head within a few days of starting to work at my first bank. Here are some of the obvious ways that software development in this particular FSC were bad.
Multi-vendor outsourcing All development had been outsourced. Not to one offshore systems integrator, but to three different ones, who are all competitors (or are they – that’s a different post). These developers were then surrounded by consultants and subject matter experts from at least four other companies. In a move that seems like the dumbest thing in the world to me, but I’ve now seen it maybe half a dozen times, testing was outsourced to a different offshore integrator.
Analysts were a separate team - or two Requirements were gathered by separate teams of analysts. In several cases, this was actually two separate teams - a team of “Business Business Analysts” (I had to check to make sure I hadn’t made this term up - it was often shortened to “Business BA” to cloak some of the strangeness) who then handed them over to a set of “IT Business Analysts”. A lot of these BA’s (ITBA’s and BBA’s) were on contract and didn’t really know what they were doing. [1]
An Embarrassment of Project managers There were at least three groups of people (actually no four) doing project management.
On “the business” side, there was a business programme manager (even though he was only managing one project) and he had several project managers working for him.
On the development side, there was a “programme manager” (who was really a project admin assistant) working for the offshore SI and a poor benighted “Scrum Master” (yes this was supposed to be an Agile project).
In between the development and the business there were two entirely separate layers of project management: the delivery project manager who liaised directly with the Scrum Master[2]; and a group of “IT Project Managers”.
IT Project Managers. I’ve seen these in every financial organisation that I’ve worked in, and I’m utterly bewildered about what it is they do. Their main role seems to be to prevent communication. To prevent communication between the developers who are doing the work and “the business” that wants the work done. Periodically they will insist that only “Business Business Analysts” can talk to “the Business” and that they are the only ones who can communicate to the IT BA’s who then talk to the developers (of course most BA’s do their damnedest never to speak to a developer and communicate only by Word documents). Another important role of the ITPM’s seems to be to obfuscate any understanding, certainly by anybody in the “the business” about the progress of the project. This is almost exactly the opposite of what Fred Brooks describes as the role of a project manager in The Mythical Man Month. In different financial organisations, they have different names - but in every one I’ve worked in, and there same stage in the process there is the same flinty-eyed phalanx of PM dark matter. One of their main roles seems to be to obscure from the business how much the project is costing; when the project is likely to finish and what progress is on the project relative to where they’d like to be. One of the most useful tools in this process is the Microsoft Project Plan. Once even a simple Microsoft Project Plan is created it’s almost unmaintainable. But once you’ve sat someone down and written a plan of over 200 lines (and I’ve seen some that are 1000 lines) then you are pretty much guaranteed that nobody is going to have any idea whatsoever what is going on. What such plans seem to be particularly good at obscuring is the things that I would like to know about any project: 1) How much work have we done so far? 2) How much work is left to do? 3) When does that mean we’re likely to finish?
OK - I can fill in more of the details about how bad it was later. The idea of this blog is that it’s going to be a commonplace - so I can throw thoughts down now, and sort them into something more coherent later. One thing that I’ve been thinking about is Coase’s Theory of the firm and Williamson’s theory of hierarchies and markets (sorry, this is a bit left-field, but trust me, it’s relevant). I realise writing this is that I really need to know what Coase’s theory of the firm is. I know more about Williamson’s theories because I’ve read his book. Williamson’s idea is that firms form because they are much better at risk reduction than markets of individuals. Firms are very efficient relative to markets of individuals.
One counter-intuitive implication of this idea is this: imagine you have a firm - it’s a really big firm so it collects all the advantages that accrue to large firms, so it’s very efficient. Here’s the question - what would you expect to see inside this firm? Efficiency? You think so? This is what I’d expect to see inside this kind of firm - massive amounts of inefficiency. Here’s how I’m starting to see it. The structure of a big firm is efficient, and this efficiency comes from it being able to essentially spread bet (this comes back to something that I was talking about in the Summer - the importance of having sociopaths who are willing to say “fuck it”). I think one of the things that Williamson would say is that one of the reasons that hierarchies (i.e. firms) are more efficient is that internally they don’t spend so much time negotiating a price (which makes multi-vendor environments interesting). So because the external structure is efficient, what you see inside is a system which is as inefficient as it can possibly be without breaking the company.
Once you start to think in this way, you stop thinking about software development in organisations as a series of lines and boxes and start to see it as a blocked stream with various bends and damns. You start to think about the efficiencies as not lines and boxes but ratios and thresholds. There is a rate of work that flows through this system that is just slow enough to keep everybody employed and just fast enough not to cause disaster. And that rate of work isn’t set by anybody - it emerges. Depending on which organisation you’re in, or which part of an organisation you’re in - all the parameters for what’s slow enough and what’s fast enough will be different.
So what I’m saying is there’s a line but it’s a line that emerges. There’s an equilibrium. And sometimes it gets crossed. Want an example? Let’s take a company I can name because I’ve never worked with them - Royal Bank of Scotland. Sometime last year their payments system broke. It broke so bad that customers of one of its subsidiary payment systems in Northern Ireland for a WEEK. Can you imagine in the 21st century not being able to get access to your bank account for a WEEK?
I was talking with a guy who worked doing testing on the internet banking for another bank. I was asking him how this disaster had come about. He shrugged “They outsourced some of the software maintenance to one of the offshore SI’s.”
There was a ‘routine’ software upgrade - the guys who would have done that before it was outsourced would have tested it before they put it on a live system. Probably they would have tested it thoroughly before they put it on a live system, because they’d been burnt before. These new outsourced guys - didn’t test enough before they let it go live. It’s in the nature of transaction systems that they cascade. System 1, talks to System 2 and 3 which talk to System 4, 5, 6, 7, which then talk to 8, 9, 10, 11, 12, 13, 14 and 15. The guys who used to deal with this system as a result of (probably bitter) experience would be watching system 1 like a hawk and would have probably caught any problems in Systems 2 and 3. The new guys didn’t know what they were looking for (they were new, and the idea that you can hand over ‘knowledge’ about how manage these kinds of things in Word documents written by disgruntled employees who are about to lose their jobs is hilarious).
There’s the line - right there. There’s as bad as it can be - and just a little bit better than that? That’s is as good as it gets.
And this is the question that I have coming out of writing all this down.
Am I an idiot?
Why do I think I might be an idiot? Let’s leave aside that in the last five years I’ve voted in zero political elections and three Eurovision song contests. Now that I’ve outlined how I think software development barely works inside big financial organisations - by being as inefficient as it can be - what possible good could I possibly imagine could come of trying to improve it? All of the mysterious forces that make firms more efficient and powerful than individuals are against me. Isn’t a lot of what I’ve been trying to do when I’ve been consulting on “Agile transformations” just totally Cnute-ish (read it carefully, I meant as in trying to hold back the tides).
What makes this even weirder is that some of these financial organisations might be totally efficient from the Coase/Williamson point of view i.e. they can never fail because if they do the government bails them out. Is the corollary of this that internally they can be nearly 100 percent inefficient? Maybe they can, as long as they keep the ATM and bank account lights on, because the unthinkable scenario of the ATM’s not working is why the Governments are prepared to bail them out. And is an even weirder twist to this that they have to keep the ATM lights on - even though they make absolutely no money out of that system and have to resort to selling poor value (i.e. fraudulent) financial products like payment protection insurance to turn a profit?
Is it just foolish to try to increase the knowledge and efficiency of an organisation which is - because of all the things known and unknown that make bigger organisations more externally efficient than markets - tending towards a certain level of ignorance and inefficiency?
I got to see the full horror of this in one “introduction to Agile” course that I sat in on in one client financial organisation. The course was run by an experienced colleague of mine a very determined lady - let’s call her Mary. As the delegates turned up it emerged that about half of the delegates knew each other. We reached the time that the course should start and so Mary, being the professional that she was, started. This resulted in an outcry from half of the assembled delegates because a lady called Liz had not yet arrived. It turned out that course we were giving that day was - in their mind at least - entirely for the benefit of Liz. Her minions were very nervous about starting without her.
Quite why became obvious when Liz finally arrived. She was, what you might call an agenda-setter. And proceeded to try to completely re-jig the agenda, oblivious to the other half of people who were on the course, who weren’t in her team. Liz claimed to be all in favour of Agile, but just wanted to modify it with some “common sense”. Here are some gems of Liz’s common sense.
*It’s just common sense that you’d write all your requirements up front before you start development. *It’s just common sense that you wouldn’t need the BA’s to be around during the development phase. *It’s just common sense that none of her team could do the job of being scrum master - they would have to “manage” someone from the offshore SI who could do the job of scrum master.
Do you know what I find so interesting about “Common Sense Liz”? She had worked her whole working life for a financial organisation that at the height of the credit crunch managed to lose 90 percent of its depositors money. Yes, that’s NINETY PERCENT. Thus forcing this particular financial institution to be forced into a shotgun marriage by the British Government with a less-stupid and less-fortunate bank in the biggest bail-out in world financial history. But none of this had dented Liz’s belief in her own “common sense”. Part of me wishes I had her self-belief.
Ooh, ooh, I think I might have thought of a new kind of consultancy - curve gouging and a strategy for making money out of this - inefficiency faking (actually, I think some of the offshore SI’s are already doing this).
So then the question arises, “What about the Japanese?” Things don’t seem to work this way in some Japanese companies. There’s the whole thing about Toyota and continuous improvement. One idea that I like is taken from Robert Levine’s book “A Geography of Time.” That is the idea that Japan is a hybrid culture, taking some of the most useful concepts from Western culture (specifically, Levine’s notion of “clock time” and the Western concept of “time is money) and combining it with Eastern notions of time - what Levine calls, “Event time.” But outside conditions possibly made it quite necessary to be quite efficient not just outside the company, but also inside the company. OK, I’m beyond the end of my knowledge and ideas here, but that might just be where the good stuff is. The way the story is told in “The Toyota Way” is that old man Toyoda came over to the US in the 20’s and had a look at the way that the Americans made cars. He realised the thing that was making money for Ford - essentially, mass-production and economies of scale, wasn’t going to make money in Japan because the market wasn’t that big, and the Japanese market had a history of fluctuating (this is an awful lot of insight, and you wonder how much of it is actually hindsight).
So Toyota evolved the notion of manufacturing cars at the rate of need - “Just in Time”. What’s drifting through my head is how difficult this is/would be in the kinds of “multi-vendor” environments that you see inside financial institutions. Somehow you have to solve the problem of “slack” in a way that doesn’t actually clog up the routes that are valuable. I’ll admit it, I get the whole notion of value stream flow and value stream mapping, but I don’t really understand how you implement this is an environment where you’ve got absolutely adherence to the prime directive - that everybody should be busy all the time.
[1] I don’t think I dreamt this story. I talked to one of the guys who was the practice lead for the “IT Business Analysts” he said he’d been given the job of hiring 300 more business analysts to work on development projects around the bank. Even in a big city like London, there simply aren’t that many people with the relevant skills on the market, so when he’d exhausted supply, having hired 200, he just hired another hundred unqualified people who applied. His target was to hire 300 people - he was going to meet his target.
[2] On one project that I worked on there were actually two delivery project managers, a delivery project manager and a deliver project manager manque (should we say without portfolio). He’d been fired for telling too much truth about the project, and then hastily reinstated - but with little left to do other than walk the halls of the project like Hamlet’s ghost and unnerve the new delivery project manager.