The roots of agile project management

Here’s a brief history of agile project management. By brushing up on these fundamental concepts, you’ll gain insight into the challenges and problems that agile techniques are designed to resolve.

Most IT project managers and software developers are familiar with the Agile Manifesto, the foundation document of the agile movement, but most IT pros are not aware of the philosophical underpinnings of this movement. The ideas that support agile development and project management didn't spontaneously spring out of the minds of the signatories to the Agile Manifesto; the ideas are based on a history of academic studies and real-world experience. Knowing these fundamental concepts will add depth and context to any discussion of agile methods.

Let's start by acknowledging that there are accepted issues with standard project management methods. The famous Standish Group CHAOS Studies demonstrate that many IT projects fail to fulfil schedule and cost forecasts, and often fail to deliver the benefits predicted. These issues have been confirmed by various organisations, including the Department of Defense (DoD). The DoD noted that, of the US$35.7 billion spent by the organisation in 1995 for software, only 2 per cent of the software was usable as delivered. The DoD found that 75 per cent of the software developed was either never used or was cancelled prior to delivery.

Other academic research challenged common IT development and project management methods. In 1998, Harvard Business School academics Robert D. Austin and Richard L. Nolan studied large software projects. Their study, which questioned many of the fundamental ideas of IT development and project management, produced these key findings:

• The first flawed assumption is that it is possible to plan such a large project.
• The second flawed assumption is that it is possible to protect against late changes.
• The third flawed assumption is that it even makes sense to lock in big projects early.

Watts Humphrey, a respected IBM researcher, followed this study with a paper outlining his Requirements Uncertainty Principle, which asserts that:

For a new software system, the requirements will not be completely known until after the users have used it.

Hadar Ziv of the University of California followed soon afterwards with his Uncertainty Principle in Software Engineering, which states:

Uncertainty is inherent and inevitable in software development processes and products.

The connection between these ideas and the underlying concepts of agile project management should be clear. If users can’t foretell what they’ll want until they see it, if predicting and planning substantial IT projects is not possible, and if protecting projects against changes that arise during the development process is impractical, the ideas behind existing "waterfall" methods are clearly flawed, and an incremental, prototype-based methodology could offer substantial benefits.

The rise of the internet ushered in a wildly innovative and experimental atmosphere in IT. Alan MacCormack, assistant professor at Harvard Business School, and two of his colleagues surveyed the software development methods of innovative internet companies. In 2001, MacCormack’s influential article Evolutionary Model of Software Development Methods outlined a history of IT development techniques, which include these models:

• Waterfall: follows a sequential process and maintains a document trail.
• Rapid prototyping: creates a disposable prototype which is exposed to the sponsor to establish customer preferences.
• Spiral: delivers a series of prototypes that incrementally incorporate user requirements.
• Incremental or staged delivery: delivers a system to customers in chunks of functional programs that are integrated incrementally to create a complete system.
• Evolutionary delivery: offers an iterative approach in which customers test an actual version of the software.

Simply recognising problems with existing methods does not solve them. In MacCormack’s article about internet companies, he recommended a set of practices that he believed could begin to replace the traditional methods. These simple precepts (which will be familiar to anyone who has researched agile project management ideas) have been cited as launching the movement towards agile techniques:

• Early release of evolving design and code,
• Daily build of code and fast turnaround on changes, and
• Deeply skilled teams.

The Agile Manifesto was the culmination of these new theories and approaches. Written in 2001 by a group of advocates of iterative and incremental development methods, this simple statement is the foundation document of the agile movement, and sets forth the underlying philosophical concepts of agile project management. The signatories include many of the founding fathers of some well-known agile methodologies: Kent Beck went on to develop Extreme Programming; Alistair Cockburn became the developer of Crystal Methods and author of influential works on agile development; and Jim Highsmith has translated agile software concepts into an Agile Project Management methodology.

Understanding the academic and experiential background of agile methods better positions us to have a persuasive conversation with our sponsors and clients, and enriches our insight into the challenges and problems that agile techniques are designed to resolve. Agile methods are not simply "sanctioned hacking" as they are sometimes caricatured, but are based on solid research, demonstrating that these incremental, iterative, prototype-based methods solve problems and offer real benefits.