I was lucky enough to be asked to speak at Agile India 2006 on the deployment production line, so I popped along today to check out the opening talks. Craig Larman, Cheif Scientist at Valtech, gave an entertaining, well-researched talk on agile vs waterfall, covering the history of both and the evidence in favour of agile. For me, the most compelling argument he presented was the long list of historical figures who promoted agile practice.
Although it's news to me, it's apparently well known that Dr Winston Royce's original paper setting out the 'waterfall' model (although he doesn't use the name) is actually describing it in order to demonstrate how poor it is as a project management technique. Larman claims that even von Neumann, working on the early stages of the Mercury Project, espoused an iterative approach to development.
It is often claimed that agile is not suited to defence projects, where techniques such as critical path development are preferred. In fact, Larman has dug out Sapolsky's book on the development of the Polaris system, where Sapolsky says that critical path analysis and PERT charts were used as a smokescreen to placate senior management while engineers got on with their iterative development process. It seems that even the Space Shuttle software development was planned as 17 6-week iterations, with "continual integration" central to the process (ACM 1984 vol. 27 issue 9).
Larman has even tracked down the source of most government directives requiring subcontractors to follow waterfall project management processes: Department of Defence directive 2167. It seems that this directive infected European government via NATO, but has subsequently been discarded by the US DoD after it lead to the failure of over 50% of their projects.
According to Larman it comes down to this: waterfall style development methods are taken from manufacturing, and are entirely acceptable when planning predictable development processes such as building cars or prefabricated houses. However when performing new product development (the example Larman gave was the pharmaceutical industry), agile processes should be preferred, and indeed have been in use for just as long. Hence inasmuch as building software is like new product development (and for the vast majority of large projects it is), agile should be preferred.