Version - 0.1 - Aristotle
The Agile manifesto is gentle in its suggestions that we should think about some rather than all. Even so, the impact of those suggestions is powerful and the method that has allowed development teams to implement the ideas in the Agile manifesto most widely and effectively is Scrum.
Scrum is more direct than the Agile manifesto. Scrum recommends four meetings: the daily Scrum or stand-up meeting; the planning meeting; the show and tell; and the retrospective. Scrum also recommends three roles for the team: Scrum Master; product owner; and team member.
Beyond that, Scrum just requires just a couple more things: that the work going to be done by the team is collected in a list called the product backlog; and that the team should be between 5 and 9 people. Teams bigger than that, should be broken down into multiple teams that are about that size.
The most obvious and noticeable thing about Scrum is the daily stand-up meeting or daily Scrum. This is the aspect of Agile that people who are only tangentially connected with Scrum teams notice and remember because it is different from normal office behaviour.
The meeting is called a stand-up because it is held standing up, less comfortable than sitting down, and so less likely to run on. In cases where a team is distributed across several locations, it is generally still a good idea for the members of the team that are in an office to stand up.
The daily Scrum is a powerful way of bringing “some” into the software development process. Each of the members of the development team are asked only to answer the following questions:
The Scrum/stand-up meeting leads the development team to focus on what is happening right now and in the very immediate future. A meeting that is supposed to be narrowly focused is vulnerable to that focus being lost. When a team member mentions that she is having problems getting access to a particular database system, it’s natural for other team members to try, in the stand-up meeting, to suggest possible solutions to her problem. It’s the job of the Scrum Master to try to keep the stand-up on track by suggesting that anyone who thinks they can help can talk to her immediately after the stand-up.
Similarly: if there are big political events going on in the world; if someone’s cat is ill; if someone’s national football or ice hockey or rugby team have won something; it’s likely that people might want to chat and joke about that rather than talk about work. Again it’s the job of the Scrum Master to try, as gently as possible, to manoeuvre the conversation back to answering the three questions.
The stand-up is a physical act, not just a psychological one. Another requirement of the daily Scrum is that it happens every working day in the same place at the same time. A slightly unusual, formalised, physical movement, done at the same time, in the same place every day, with quite strict rules as to how it should be conducted? Without pushing the analogy too far, it’s possible to see that the daily stand-up is a bit like a religious ritual, or military ceremony: like roll call; morning prayer; or a march around the parade ground.
So the daily stand-up is a slightly unusual physical act which aids a slightly unusual, for project management, psychological act: that of focusing on what is happening right now, rather than on the whole of the project. Even so, this brief, change in behaviour and thinking might be the most important and powerful aspect of Scrum.
In sprint planning, which we’ll talk about in a minute, the product owner, the product backlog and the development team come together to achieve a shift in focus from the “all” of traditional Waterfall planning to the “some” of Agile planning.
As we mentioned above, the product backlog is a list of the things that need to be done on a project. It might be tempting to think that this is very similar to the requirements and specifications documents that are written in a traditional waterfall project. But the product backlog is different from a traditional specification document in at least three ways.
The product backlog differs from a traditional requirements document in that it is a collection of pieces rather than a single whole. These bits that are collected together in the product backlog are referred to as “stories”. We don’t have to worry just now about the precise details of what constitutes a story. What’s important is that the backlog is a collection of individual items rather than a single, monolithic document. This allows the second major difference from a specification document: deliberate vagueness and incompleteness.
Not everything in the product backlog has to be fully detailed. Items which are currently a low priority can have almost no detail at all, they can also be left in big chunks. Conversely items that have been give top priority, do need a substantial amount of detail in them and do need to be broken down into smaller bits.
That items in the product backlog which are not top priority don’t need to be completely detailed makes the product backlog more useful than a traditional specification. Even the items at the top of the backlog, those with the highest priority, don’t need to be fully and completely detailed. This is useful because it stops stories getting held up simply on the grounds that they aren’t “complete”. What Scrum is explicitly admitting is that further work and discussion will need to go on about requirements when those stories are being worked on by the development team.
One of the main reasons that the product backlog is collected in bits is so that they can be ordered in different ways. This is one of the main jobs of the product owner: to prioritise the items in the product backlog.
By doing this the product owner becomes the role in Scrum which conjures most powerfully with “some” and “all”. All of the work that might be done on a Scrum development project is kept in the product backlog. But one of the most powerful aspects of Scrum is that at the beginning of every sprint the product owner needs to decide on some of these things that should be planned to be worked on by the development team.
Most product owners find this extremely difficult, at least at first. Looking at things from the point of view of symmetry and asymmetry we get an idea of why. The act of prioritising what needs to be done is “all destroying.” Once we break ground on a project and actually start to do it, there’s a strong possibility that we will find problems with the vision. Once we start to do a project, there’s a strong possibility that we will realise that the whole of the project could take much longer than was initially predicted.
Again, it is often the Scrum Master who is left with the job of persuading the product owner to articulate which items should be a priority.
One strange thing about this resistance to the prioritisation of the product backlog is that often people know which bits of the product backlog are the most important and should be the priority.
I worked on a fraud detection project that was estimated to take nearly two years. Even so, the bit that everyone knew was most important was an algorithm that detected false positive indications of fraud. This was perhaps only five percent of the whole project and could have been delivered in just a few months.
It can be useful to ask the product owner which small part of the project it is that everyone knows should be done first. Or having talked to other people who know about the project, the Scrum Master might suggest this as the bit that should be given highest priority.
Another way to help the product owner is to explain that even though everything in the product backlog may need doing, the development team can’t do all of the things in the product backlog at the same time. So the question of prioritisation then becomes a slightly easier question: even if we agree that we’re going to everything in the product backlog, where should we start?
Once the team has started, there’s then an obvious, easier follow-up question than explicitly pushing for prioritisation: what should we do next?
Scrum mentions just four meetings, but often there is an extra meeting which is held once or twice during each sprint. This is a meeting where the product owner talks through with the team what is coming up next.
The aim of these sessions is to make the stories a bit more detailed. Detailed enough that they’re ready for development. But there’s a symmetrical danger here, a danger of looking for all rather than some of the detail. Teams have to get the hang of knowing when a story is detailed enough that it can be planned, otherwise they get stuck. Again, part of job of the Scrum Master is to spot when the team are getting too hung up on detail, or decide that a story has enough detail and move on to discuss something else.
In Agile and especially in Scrum, rather than trying to get an exact idea of how long things are going to take, we try to get a rough idea using complexity estimation. This is done by estimating how complex a story is relative to other stories: if a story is the same complexity as another story we give it the same complexity score. If it’s more complex we give it a higher complexity score and so on.
This means that when we get to the end of a sprint we will have a list of stories that we’ve finished and a count of complexity points. This gives us a rough idea of how many complexity points we could manage in the next sprint.
If the product owner has managed to prioritise the backlog and some refinement has been done on these stories, sprint planning should be a fairly painless exercise.
The team and the product owner go through the stories which are top of the product backlog, discuss them and plan them in priority order until all are agreed that they have planned enough work for the next sprint (sprints are typically two weeks long).
Note that this is an agreement between the team and the product owner. The product owner isn’t telling the team what to do. The team and the product owner are agreeing what to do. How do the team know how much work will be enough for a sprint? The team uses how much work was done in the previous sprint, estimated in complexity points, as a guide.
Again, the role of the Scrum Master is to spot when either the team have taken on too much; when the product owner is pushing the team to take on too much; or when there is still some room to plan more.
Scrum dictates that at the end of every sprint there should be a meeting where the team demonstrate what they have done in that sprint. For some reason no one seems to have settled on name for it, it’s called either the “Sprint Review”, “Show and Tell,” or the “Showcase” or the “Sprint Demo.” Of all of these names, “Show and Tell” appeals to me most, so it’s the one I’ll use.
My own experience is that the “Show and Tell” can be one of the most difficult meetings in Scrum and the reason is because of this relationship between “some” and “all.” In the show and tell, the people who are paying for the project get to see what they’re getting for their money. Certainly in the early days of a project, this isn’t much.
The people who are looking at this “working software” are the people who have bought into this project because of some dream: some big, simplified, easy to state and attractive “all”. Sometimes being faced with the gap between what has been promised and what needs to be delivered, is difficult.
That is possibly why getting the people who need to come to the showcase: senior stakeholders and even the product owner can be hard. A lot of the time they won’t like what they see, or they won’t feel that what they see is near enough being a finished product to be worth their attention. Often, like the joke about the elderly ladies complaining about their stay in a hotel, not only is the food terrible, it’s in such small portions.
Again it often falls to the Scrum Master to ensure the continuing engagement between the development team and the stakeholders - offering some explanation of what the Product Owner is actually looking at and also explaining to stakeholders what the value is of some small piece of working software.
So what should stakeholders do in these circumstances? For a long time, I thought what they should do is come to all the meetings and take the extremely slow progress on the chin, give detailed and definitive feedback and most of all keep their nerve.
Since, then I’ve started to reflect on the product owners I’ve seen who’ve really worked well, and they don’t do that. There are two modes of product owner that I’ve seen really work: stealth and what I’m going to describe as “anti-Pharaohs on elastic.” (OK, this needs a better name).
On a project where are there are lots of big beasts (i.e. important stakeholders that can’t be argued with) the product owner provides good solid guidance about priority and good feedback to the team on what they are producing, but never confronts the big beasts. This means that the shiny “all” goal and dream of the project and the unreality of deadlines is never challenged, nor some of the fundamental obstacles to the product’s success are never addressed or removed. Still when the big beasts aren’t around, the product owner does what they can. This isn’t a perfect arrangement, but it can work and indeed can be the only possible solution where product owner is too junior to stand up to the big beasts, or in an environment where any discussion of the real state of the project is likely to result in dismissal.
The Pharoah in book of Exodus in the Bible asked the children of Israel to make bricks without straw. Pharaoh wasn’t an idiot, he knew what he was doing, he knew that making bricks without straw was impossible and he did it as a punishment.
Unfortunately, the Pharaohs that we encounter on modern projects often don’t know that they’re asking for the impossible. They ask for software that talks to a database that doesn’t exist; they ask for software using a technology that doesn’t work properly yet; or they ask for something, but totally refuse to give the requisite detail of what it is that they want.
One of the things that Scrum does is that it acts as a way of testing hypotheses, of trying out ideas like “We can connect to this database” or “this cool new technology actually works” and we’ll talk about this in the next chapter.
By breaking the work up in to small chunks and trying to do those chunks in a short space of time it soon becomes obvious what the problems are with a project. And quite a few of these problems are “bricks without straw” problems. They are problems that if they aren’t solved will mean that the project has no chance of success.
A good product owner behaves in ways that might seem contradictory, they behave in an “elasticated” fashion. When things are going badly, they keep themselves at a slight distance from the project. But they also pay attention to the demands of the team for help with their bricks-without-straw problems. They stay sufficiently in-touch that if and when the project does start to succeed, they can snap back and be much more involved. When it comes to taking the credit, they are of course front and centre stage.
So, although the stealth product owner and the elasticated non-Pharoah product owner might seem to be behaving very differently, their attitude to the product is quiet similar. One way that you might talk about the successful approaches of both the stealth product owner and the elasticated product owner is that they aren’t fully invested in the project. Another way of saying this is that this isn’t an “ego” project. This kind of language highlights two all ways of looking at a project which are extremely damaging.
“Fully invested” suggest that someone has put all of their resources into this single project, possibly they have also put the full weight of their reputation behind a project. Given the chances of success of any software project, putting your whole reputation behind a project probably isn’t that smart a move.
An “ego project” (“ego” is the Latin for “I”) is a project that someone is directly and closely associated with. Again, this isn’t a sane way of thinking about software projects because even with well-specified, well-managed software projects, there is still a large amount of uncertainty and a chance that they will go wrong. What’s important to highlight here is that associating the ego with a project is symmetrical thinking. It’s thinking that associates success with there being a symmetry between what’s in the mind of the product owner and the software that’s produced. In very bad cases, this can also mean taking any bricks-without-straw problems that are identified in the project as personal criticisms.
The best product owners don’t do this. They have some attachment to the project (especially when it’s going well) but they do not identify with it completely and they help wherever they can to remove “bricks without straw” problems.
The retrospective is a short meeting where the team get together and discuss what went well; what didn’t go so well; and what they might do differently to improve the performance of the team.
The retrospective also enforces a some perspective because, although there might be discussion of what an ideal situation would look like, one of the main purposes of the retro is to identify specific next actions to improve the way that the team is working.
For product owners who aren’t Pharoah (i.e. they’re interesting in giving their workers the tools they need to do the job) the retrospective is also full of vital information because it gives a list of the problems that are stopping the team from doing their work.
I’ve been facilitating retrospectives now for about 8 years and from the very beginning I instinctively felt that, although it was a good idea to capture all the points that arose at the retro, it was also a good idea to boil those points down to just two or three points which would get reported more widely: just some of the issues.
If this effort isn’t made, the retrospective can easily fall prey to unhelpful “all” thinking. For instance there can be the idea that all of the actions that come out of a retrospective should be completed before the next retrospective. This can be made even worse by making each action a commitment (we’ll talk about commitment and consistency in a separate chapter).
It took me quite a while to realise that the natural cadence of resolution for actions from retrospectives isn’t the same as the cadence of Scrum sprints (which are typically every two weeks). Some issues that come out of retrospectives can be fixed quickly, some will take longer, some will be problems throughout the lifetime of a project.
In this chapter we’ve explored the way that Scrum is about “some”. In particular the meetings that Scrum recommends are a way of forcing the product owner and, by extension other stakeholders to think about the detail of some rather than the general idea of all.
But Scrum isn’t just about “some”. Scrum is also about “all”. One way that Scrum is about “all” is that it is absolute in its partiality. For example there is the insistence that the Scrum happens every working day at the same time, in the same place and that no other questions are answered other than “what did I do yesterday”; “what am I going to do today”; “what’s blocking me?”.
Scrum also focuses on completeness. The aim of the Sprint is to complete all the work that was planned in the Sprint and that work is not counted as being “Done” until the working software has passed its tests and it has been accepted by the product owner.
A substantial part of the power of Scrum is in its approach to attacking symmetrical thinking and replacing it with asymmetrical, rational, logical thinking. But, Scrum being a human endeavour, it is in itself vulnerable to inappropriate and unhelpful symmetric thinking. Below are some of the ways in which Scrum can end up insisting on “all” where it would be better to think more carefully about “some”.
Scrum works very well for software development, where once work is started on a product backlog item, it usually can continue until the item is done. Scrum is less well-suited to managing other kinds of tasks.
I’ve been involved in many Agile coaching groups inside large organisations. To my shame, many of these groups have been seduced by the idea “we’re advocating Scrum to our clients, we should probably use it ourselves.” Very often this gets as far as planning in sprints, but it misses out some of the things that are really important about Scrum. For example, such attempts at using Scrum to manage consultancy-level work often remove,the main engine of some in a Scrum team - the daily stand-up. They sometimes also miss appointing either a Scrum Master or even a product owner and then wonder why nothing gets done.
This is in direct contradiction to the idea that stories have to be just detailed “enough.” The reason that we don’t insist that they are completely (all) detailed is that this is impossible and the insistence stops the backlog refinement process dead in its tracks.
Sometimes, some breaking down of stories into more detailed tasks is useful. But insisting that all stories be broken down into the smallest level of detail brings us back to the waterfall problem of not knowing what the standard is for “completely detailed.”
The false idea that everything can be broken down into smaller and smaller tasks is also possibly an attempt to deny one of the most important truths about software development: it is knowledge work. It requires the solution of problems by smart people and quite how those problems will be solved isn’t something that can be broken down into discreet steps.
One of the “Scrum values” which is outlined in the admirably brief “Scrum Guide” is commitment. Commitment and consistency is a powerful psychological principle, which aligns well with “all-ness thinking” and “symmetry”. In the introduction, we’ve already seen how dumb this kind of thinking can be and it’s such a powerful way of behaving badly using symmetry that we’ll devote a later chapter to it.
From the point of view of “some” and “all” it’s interesting that if too much emphasis is placed on the idea of completion of everything that was planned for the sprint, “all”-ness can spread. The development team might begin to insist that stories that are committed to in planning are completely (“all”) detailed. When any vagueness or uncertainty is unearthed the team may down tools on the stories; mark them as blocked; and stop work on them until all uncertainty is removed. They might also disavow any committment to a story which they said they would do, but which then turned out to be “incomplete.”
So what does the Scrum Master do? One way to think of the Scrum Master is that they are the keepers of the “some” perspective. The Scrum Master is in many ways the “some” master (oh dear). In the daily stand-up they try to keep the team focused on what happened yesterday, what’s happening today and anything that’s immediately stopping that from happening. They keep the focus on the small picture.
In planning and backlog refinement, they help the team and the product owner agree to take on some of the work from the product backlog. They also reassure the team and product owner that the work they take on does not need to be completely (all) detailed, but can be planned when it is detailed enough (some).
During the sprint, they help the team focus on getting some of the work done but here they also focus on the absolute requirement of Scrum to only count work as done when it has produced working software.
In this chapter I’ve looked at the ways that the Scrum method encourages, even forces a “some” perspective on the work that needs doing in software development. This focus on some is powerful for at least three reasons.
The stasis and paralysis that can afflict projects that insist on all requirements being completely ready before they can proceed is avoided by the willingness of Scrum to start working on some of the requirements when they are good enough. This perhaps the most powerful aspect of Scrum.
The focus on getting some of the work in the product backlog working produces working software which could potentially be valuable to the customer: providing customer feedback; clarifying in the customer’s mind what it really was they they wanted; and delivering revenue to the organisation.
By trying to do some of the work in the product backlog, Scrum provides us very quickly with a far more informed idea of what obstacles there might be which could prevent us from doing what we want to do.
By doing this Scrum is also checking the vision and the dream of “all” of the items which constitute a project against the “some” of what has actually been achieved. In this way Scrum is a method for making explicit and testing the hypotheses about the project which are implicit in the work in the product backlog and it’s this hypothesis testing aspect of Scrum that we’ll look at in more detail in the next chapter.