There was this time, known as the bad old days, where programmers were seen as the assembly line workers and mechanics. The perception was that programmers didn’t create, they were the people who merely put together and fixed software systems, systems that were really ‘created by’ people who wrote the specifications.
Giant requirements documents were dumped onto programmers desks, then they were told that most of the work had been done already and all they had to do was put it together. Then after slaving away building a system for months or years, they would find the thing they created was not what the customer wanted, and was essentially useless and/or fundamentally flawed, e.g. INCIS or Novapay.
I did not work in that bad world, but I have heard stories from elder programmers, who, with a glazed over thousand-yard stares, describe their experiences with the waterfall process, requirements documentation and TPS reports.
With newer, better agile methodologies of developing software, there is now more room for programmers become the creators, and masters of their domain. Where the user is the centre of the process and the programmer is building what they actually need.
In this post, I am going to briefly look at Waterfall, Scrum, Extreme Programming, Agile, and Kanban. This is not an in-depth examination or comparison between them, but a quick introduction to their history and concepts.
Why use Software Methodologies?
As programmers we create and maintain software. The process by which we do this will determine the outcome of the code. Will the code we write be good and the project a success, or will the code be broken, and the project be over-budget, unmaintainable and not needed by the user?
To increase the chances that a software project will be successful, previously successful practises were grouped into Software Methodologies. Basically, these are sets of good ideas that were formalised into methods to help developers build successful software.
1970 — Waterfall
Royce, when first describing Waterfall wrote that it is:
risky and invites failure.
From the outset Waterfall was seen as a bad idea.
Waterfall proposes that you develop software by going down through phases of development: Requirements, Design, Implementation, Verification, and ending at Maintenance.
The idea being that a software system is defined and refined at each stage, and a document is passed to the next stage to complete their part. The core benefit of this approach is that spending time upfront on requirements and design will save more money later as solving problems earlier is significantly more efficient.
However, the main criticism of Waterfall is that customers and users will not know what they want upfront. They do not know or understand what is possible or feasible or the best way to get either. Software instability can also be due to a feedback cycle (Lehman describes Laws of Software Evolution), where once a user has a working system it effects the environment the system is deployed in, thus changing and adding requirements.
Additionally, as stated by Steve McConnell in Code Complete, design requirements and limitations cannot be entirely known before completion. Where following Waterfall means all the requirements and design being done before any implementation is done.
Waterfall has fallen out of favour in recent times, though it is still used for many projects. A good discussion on some of the reasons why it continues being used, particularly in government, is given in the Ruby Rogues podcast Ruby in Government with Sarah Allen.
1995 — Scrum
Although the ideas in the Scrum methodology have been around since the 1980’s, it was not yet formalised and presented until OOPSLA in 1995 by Jeff Sutherland and Ken Schwaber. The book Agile Software Development with Scrum written by Schwaber and Beedle was later published as the core work defining the Scrum methodology.
Scrum is an iterative approach to development. That is, Scrum quickly repeats the same steps until the project is finished. This is in response to the often changing mind of the user, and is used to increase the ability to quickly adapt to changing requirements.
Features are written from the perspective of the user and are called User Stories. All the user stories are combined into the product backlog.
The product owner is the person who decides what goes into the product backlog and they decide product’s direction as they represent the customers and users of the product.
The scrum master (basically the manager) is there to make sure the project moves smoothly and that everyone has what they need. Also, they set up meeting and facilitates release planning.
The developers build the product by implementing the user stories.
The testers test the product to make sure the user stories are completed correctly.
The customer uses and (hopefully) pays for the product.
Release Planning is where the team identifies stories from the product backlog and moves them to a release backlog. There the stories are prioritised and the amount of work to implement them is estimated. If a story is too large it is broken down into smaller stories. By summing up all the estimates from the stories the estimate of the total work required for release can be given.
There are many ways to estimate the work amount, a common way is to play a game called Planning Poker (described in Agile Estimating and Planning) where each player is given a deck of Fibbonaci numbers (0, 1, 2, 3, 5, 8, 13, 21…), the large difference in the higher numbers is a reflection of how difficult it is to estimate large values. Each story is discussed then the each person in the team picks a card to estimate the amount of work they think it will take. The members that pick the highest and lowest numbers get to discuss why they picked them, then the process is repeated till a consensus is reached.
With the prioritised and estimated user stories the sprints can now be planned. Sprints are short-duration milestones that allow teams to tackle a manageable chunk of a product, and get it to a ship ready state. Sprints can range from a few days to a month, and there are two to a dozen sprints in a release. The release backlog is split into the sprint backlog, where the goal of each sprint is to get a set of user stories that will be ready to ship by the end of the sprint.
As the sprints progress it is very important to monitor the progress of the stories. This is because the measurement of the teams progress will show if the project is behind schedule. This is where the burndown chart comes into play.
The burndown chart is a day by day measure of the remaining work in a sprint or release. This is calculated each day as developers update the stories they are working on with the estimated amount of effort remaining till completion.
The total amount of work remaining should be trending downwards as stories are completed. The rate at which it is falling, the slope of the graph, is known as the velocity, which is the average rate of productivity of the team per day. Knowing the velocity allows the team to calculate the estimated completion date for the sprint or release.
The burndown chart is a large reason for the popularity of scrum as it is a great visualisation tool to show the overall progress of the product. It is accessible by all non-technical and technical team members, and it conveys a massive amount of data in a few seconds. It is also useful in seeing the progress being made early on in the project, and enabling the team to compare actual velocity with needed velocity and make changes if necessary.
The daily scrum is a communication tool that is used to let information flow freely between team members. It is a fast paced standup meeting where people list the work they have completed and blocking obstacles they are facing. It lets major issues come to light and be dealt with quickly.
At the end of each sprint there is a sprit retrospective that lets everyone discuss what went right and where to improve. This allows for the method to be altered iteratively till it fits the team best.
1999 — Extreme Programming (XP)
In 1996 Kent Beck became lead of the Chrysler Comprehensive Compensation System (C3) payroll project, which intended to replace several independent payroll systems with one. While managing this project Beck refined the development methodology that would eventually become extreme programming. This resulted in Beck writing Extreme Programming Explained in 1999.
Extreme programming is called as such because it takes its practises to the extreme, like always writing tests before code in test-driven development.
The introduction of Extreme Programming had a significant impact on the software development landscape. It was especially notable for its emphasis on test driven development practises, with things like ‘Keep it simple stupid’ (KISS), ‘You aint gonna need it’ (YAGNI) and ‘Fake it till you make it’.
Some of the methods that XP prescribes are seen as too extreme by many, and often people reduced or removed some of the practices. The overall effect was that XP’s many practices have been widely adopted and are followed if not to the extreme nature they were originally intended.
2001 — Agile
Many iterative software methodologies like Scrum and XP were now out in the wild being used by developers. They had similar core philosophies and ideas like adapting to requirements and allowing for change. In February 2001, 17 developers meet to discuss their development methods and out of this meeting came the Agile Manifesto:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
This broad set of guidelines was created and signed by many influential people including Kent Beck (creator of XP and Test Driven Development), Ken Schwaber and Jeff Sutherland (creators of Scrum), Martin Fowler (author of [Patterns of Enterprise Application Architecture] (http://www.amazon.com/gp/product/0321127420/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321127420&linkCode=as2&tag=maor01-20)and Refactoring: Improving the Design of Existing Code).
More than 10 years later the agile manifesto is still an important document in the world of software engineering. Although software has rapidly advanced over this time, the guidelines that Agile prescribes are still applicable. This demonstrates that as technology changes the nature of programming and development do not. This is one reason why understanding methodologies is an important aspect of software development.
The Agile Alliance was formed by some of the signees with the goal of supporting:
those who explore and apply Agile principles and practices in order to make the software industry more productive, humane and sustainable.
2003 — Kanban
In September 2003 the book Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results described the Kanban System (or the Toyota System) to organise and deliver work.
You start with a board that has lanes with titles such as ‘Backlog’ and ‘In Progress’. Then you represent user stories as cards and place them in the lanes from left to right as to represent their completion. This way you can quickly gauge where there is a backup of stories on the board. The number and titles of the lanes are up to you, but the simpler the better.
To help ensure items are being completed each of the lanes can have a work-in-progress or WIP limit, that is the limit of the number of cards in a single lane. When there are too many cards you can easily identify the problem and then take steps to find a solution.
Kanban is more about getting-things-done rather than an entire software specific methodology. It can be used within other methodologies like Scrum to help progress through stories.
Similar to Kanban is the Lean Software Development (LSD) method described by Mary and Tom Poppendieck in
Lean Software Development: An Agile Toolkit. It also takes inspiration from the Toyota but contains more concepts that are software specific.
Conclusion
These methodologies are tools for your belt. However, they are all still evolving (even Waterfall) in the way they used. So like any tool, they must be kept sharp. Adding tools to your belt is also necessary to dodge the trap where “every problem is a nail, when I only have a hammer”.